Skip to main content
Open In ColabOpen on GitHub

AgentQLLoader

AgentQL's document loader provides structured data extraction from any web page using an AgentQL query. AgentQL can be used across multiple languages and web pages without breaking over time and change.

Overview​

AgentQLLoader requires the following two parameters:

Setting the following parameters are optional:

  • api_key: Your AgentQL API key from dev.agentql.com. Optional.
  • timeout: The number of seconds to wait for a request before timing out. Defaults to 900.
  • is_stealth_mode_enabled: Whether to enable experimental anti-bot evasion strategies. This feature may not work for all websites at all times. Data extraction may take longer to complete with this mode enabled. Defaults to False.
  • wait_for: The number of seconds to wait for the page to load before extracting data. Defaults to 0.
  • is_scroll_to_bottom_enabled: Whether to scroll to bottom of the page before extracting data. Defaults to False.
  • mode: "standard" uses deep data analysis, while "fast" trades some depth of analysis for speed and is adequate for most usecases. Learn more about the modes in this guide. Defaults to "fast".
  • is_screenshot_enabled: Whether to take a screenshot before extracting data. Returned in 'metadata' as a Base64 string. Defaults to False.

AgentQLLoader is implemented with AgentQL's REST API

Integration details​

ClassPackageLocalSerializableJS support
AgentQLLoaderlangchain-agentql✅❌❌

Loader features​

SourceDocument Lazy LoadingNative Async Support
AgentQLLoader✅❌

Setup​

To use the AgentQL Document Loader, you will need to configure the AGENTQL_API_KEY environment variable, or use the api_key parameter. You can acquire an API key from our Dev Portal.

Installation​

Install langchain-agentql.

%pip install -qU langchain_agentql

Set Credentials​

import os

os.environ["AGENTQL_API_KEY"] = "YOUR_AGENTQL_API_KEY"

Initialization​

Next instantiate your model object:

from langchain_agentql.document_loaders import AgentQLLoader

loader = AgentQLLoader(
    url="https://www.agentql.com/blog",
    query="""
    {
        posts[] {
            title
            url
            date
            author
        }
    }
    """,
    is_scroll_to_bottom_enabled=True,
)

Load​

docs = loader.load()
docs[0]
Document(metadata={'request_id': 'bdb9dbe7-8a7f-427f-bc16-839ccc02cae6', 'generated_query': None, 'screenshot': None}, page_content="{'posts': [{'title': 'Launch Week Recap—make the web AI-ready', 'url': 'https://www.agentql.com/blog/2024-launch-week-recap', 'date': 'Nov 18, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Accurate data extraction from PDFs and images with AgentQL', 'url': 'https://www.agentql.com/blog/accurate-data-extraction-pdfs-images', 'date': 'Feb 1, 2025', 'author': 'Rachel-Lee Nabors'}, {'title': 'Introducing Scheduled Scraping Workflows', 'url': 'https://www.agentql.com/blog/scheduling', 'date': 'Dec 2, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Updates to Our Pricing Model', 'url': 'https://www.agentql.com/blog/2024-pricing-update', 'date': 'Nov 19, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Get data from any page: AgentQL’s REST API Endpoint—Launch week day 5', 'url': 'https://www.agentql.com/blog/data-rest-api', 'date': 'Nov 15, 2024', 'author': 'Rachel-Lee Nabors'}]}")
print(docs[0].metadata)
{'request_id': 'bdb9dbe7-8a7f-427f-bc16-839ccc02cae6', 'generated_query': None, 'screenshot': None}

Lazy Load​

AgentQLLoader currently only loads one Document at a time. Therefore, load() and lazy_load() behave the same:

pages = [doc for doc in loader.lazy_load()]
pages
[Document(metadata={'request_id': '06273abd-b2ef-4e15-b0ec-901cba7b4825', 'generated_query': None, 'screenshot': None}, page_content="{'posts': [{'title': 'Launch Week Recap—make the web AI-ready', 'url': 'https://www.agentql.com/blog/2024-launch-week-recap', 'date': 'Nov 18, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Accurate data extraction from PDFs and images with AgentQL', 'url': 'https://www.agentql.com/blog/accurate-data-extraction-pdfs-images', 'date': 'Feb 1, 2025', 'author': 'Rachel-Lee Nabors'}, {'title': 'Introducing Scheduled Scraping Workflows', 'url': 'https://www.agentql.com/blog/scheduling', 'date': 'Dec 2, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Updates to Our Pricing Model', 'url': 'https://www.agentql.com/blog/2024-pricing-update', 'date': 'Nov 19, 2024', 'author': 'Rachel-Lee Nabors'}, {'title': 'Get data from any page: AgentQL’s REST API Endpoint—Launch week day 5', 'url': 'https://www.agentql.com/blog/data-rest-api', 'date': 'Nov 15, 2024', 'author': 'Rachel-Lee Nabors'}]}")]

API reference​

For more information on how to use this integration, please refer to the git repo or the langchain integration documentation

Was this page helpful?