i@dZddlZddlmZmZmZmZmZmZddl m Z m Z m Z m Z mZmZmZddlmZmZmZmZddlmZddl mZd eeed eefd Zd eeefd eeeffd Zddddddddddedeedee deeeefdeedeedeedeedeedeed e fdZ d1dededeed e fdZdddededee d e fd Z! d1deded!eedeed eef d"Z"deded efd#Z# d2deded$ed%eed e f d&Z$ddddddddddd' dedeedee deeeefdeedeedeedeedeedeed$ed%eed e fd(Z%deed eefd)Z&dddddddd*deedee deeeefdeedeedeedeedeed e'fd+Z(d3deed-ed eeefd.Z) d4dedeedee d-ed$ed%eed eefd/Z*deded efd0Z+dS)5z4 Batch scraping functionality for Firecrawl v2 API. N)OptionalListCallableDictAnyUnion)BatchScrapeRequestBatchScrapeResponseBatchScrapeJob ScrapeOptionsDocument WebhookConfigPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_input)CrawlErrorsResponse data_listreturncg}|pgD]E}t|tr.t|}|t di|F|S)N) isinstancedictrappendr)r documentsdoc normalizeds d/home/agentuser/.hermes/hermes-agent/venv/lib/python3.11/site-packages/firecrawl/v2/methods/batch.py_parse_batch_scrape_documentsr"sb "IB55 c4  51#66J   X33 33 4 4 4 bodyc |ds#t|dd|d|dd|dd|d|d |d t|d gpgd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextdatar)r*r+ credits_used expires_atr.r/)get Exceptionr")r$s r!#_parse_batch_scrape_status_responser5s 88I  E*BCCDDD((8$$XXk1--'1%%//hh{++  -dhhvr.B.B.HbII  r#optionswebhook append_to_idignore_invalid_urlsmax_concurrencyzero_data_retention integrationidempotency_keyclienturlsr7r8r9r:r;r<r=r>c t||||||||} || } |d| | } | jst | d| } | ds#t| ddt| d| d | d pd S) aW Start a batch scrape job for multiple URLs. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options Returns: BatchScrapeResponse containing job information Raises: FirecrawlError: If the batch scrape operation fails to start r7r8r9r:r;r<r=z/v2/batch/scrape)headerszstart batch scraper&r'r(idurl invalidURLsN)rDrE invalid_urls) prepare_batch_scrape_request_prepare_headerspostokrjsonr3r4r )r?r@r7r8r9r:r;r<r=r> request_datarCresponser$s r!start_batch_scraperO-s80 !/'/   L%%o66G{{-|W{MMH ;>h(<=== ==??D 88I  E*BCCDDD  88D>> HHUOOXXm,,4   r#job_idpagination_configc |d|}|jst|d|}t |}|d}|r|jnd}|r |drt ||d||}t|d|d|d|d |d |s|dnd | S) ak Get the status of a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: FirecrawlError: If the status check fails /v2/batch/scrape/zget batch scrape statusr/Tr.r)r*r+r1r2Nr0)r3rKrrLr5 auto_paginate_fetch_all_batch_pagesr )r?rPrQrNr$payloadrrTs r!get_batch_scrape_statusrWgs(zz6f6677H ;Ch(ABBB ==??D1$77GI8IR%33dM  *  FO      x +&g^,<($1 ;WV__t    r#)request_timeoutnext_urlrXc $|||}|jst|d|}t |}t |d|d|d|d|d|d|d  S) a Fetch a single page of batch scrape results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutzget batch scrape status pager)r*r+r1r2r.r/r0)r3rKrrLr5r )r?rYrXrNr$rVs r!get_batch_scrape_status_pager\s(zz(Oz<'>   %DN,<,cN d d d MMOO  >yIILL    E %V, ' 'H&3y>>[+H+H   X & & & &  "s9~~'D'D #6* a K %N s C00 C=<C=c|d|}|jst|d|}|ddkS)a( Cancel a running batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job to cancel Returns: BatchScrapeStatusResponse with updated status Raises: FirecrawlError: If the cancellation fails rSzcancel batch scraper) cancelled)deleterKrrLr3)r?rPrNr$s r!cancel_batch_scraperw sa$}}999::H ;?h(=>>> ==??D 88H   ,,r# poll_intervalr[ctj} t||}|jdvr|S|r0tj|z |krt d|d|dtj|b)a Wait for a batch scrape job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the batch scrape job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the job fails or timeout is reached TimeoutError: If timeout is reached T)r*failedruzBatch scrape job z did not complete within z seconds)rgrhrWr) TimeoutErrorsleep)r?rPrxr[ro status_jobs r!wait_for_batch_completionr~&s,!!J ",VV<<    D D D   g((:5@@e6eeT[eeeff f =!!! "r#) r7r8r9r:r;r<r=r>rxr[c ft||||||||||  } | j} t|| | | S)a Start a batch scrape job and wait for it to complete. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the batch scrape fails to start or complete TimeoutError: If timeout is reached r6)rOrDr~)r?r@r7r8r9r:r;r<r=r>rxr[startrPs r! batch_scraperMs]B  !/'/'   EXF % w  r#cN|stdg}|D]}|rt|tstd||ds'|dstd||||S)z Validate and normalize a list of URLs for batch scraping. Args: urls: List of URLs to validate Returns: Validated list of URLs Raises: ValueError: If URLs are invalid zURLs list cannot be emptyz Invalid URL: zhttp://zhttps://z)URL must start with http:// or https://: ) ValueErrorrstr startswithrstrip)r@validated_urlsrEs r!validate_batch_urlsrs 64555N++ 4*S#.. 42S2233 3y)) PS^^J-G-G PNNNOO Ociikk**** r#rBclt|}d|i} |r&t|} | r| | |4t|tr|| d<n|d| d<||| d<||| d<||| d<||| d <|$t || d <| S) z Prepare a batch scrape request payload. Args: urls: List of URLs to scrape options: Scraping options Returns: Request payload dictionary r@Nr8T) exclude_none appendToIdignoreInvalidURLsmaxConcurrencyzeroDataRetentionr=)rrupdaterr model_dumpr) r@r7r8r9r:r;r<r=rrM scrape_datas r!rHrHs*)..N$*N#;L-,W55  -    , , , gs # # L&-L # #&-&8&8d&8&K&KL #%1 \"&,? ()")8 %&&,? ()&)+&6&6&<&<&>&> ]# r#d chunk_sizecg}tdt||D]"}|||||z#|S)z Split a large list of URLs into smaller chunks for batch processing. Args: urls: List of URLs to chunk chunk_size: Maximum size of each chunk Returns: List of URL chunks r)rangerlr)r@rchunksis r! chunk_urlsrsRF 1c$ii , ,.. d1Q^+,---- Mr#ct||}g}d}|D]<} t|| |||} | jr|| j|dz }=|S)a Process a large batch of URLs by splitting into smaller chunks. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options chunk_size: Size of each batch chunk poll_interval: Seconds between status checks timeout: Maximum seconds to wait per chunk Returns: List of all scraped documents Raises: FirecrawlError: If any chunk fails r)r7rxr[rb)rrr/extend) r?r@r7rrxr[ url_chunks all_documentscompleted_chunkschunkresults r!process_large_batchrs2D*--JM  '     ; .   - - -A r#cP|d|d}|jst|d|}|d|}|dg|d|dgd}t d i|S) z Get errors for a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job Returns: CrawlErrorsResponse with errors and robots-blocked URLs rSz/errorszget batch scrape errorsr/errors robotsBlockedrobots_blocked)rrr)r3rKrrLr)r?rPrNr$rVr s r!get_batch_scrape_errorsrszz=f===>>H ;Ch(ABBB ==??Dhhvt$$G++h++!++ow{{CSUW7X7XYYJ  , , , ,,r#)N)r N)r)Nrr N),__doc__rgtypingrrrrrrtypesr r r r rrrutilsrrrrutils.normalizerrr"rr5boolintrOrWfloatr\rUrwr~rrrrHrrrrr#r!rs_ ================gfffffffffff666666''''''Xd3i-@T(^ d38n c3h    &(,37"&*.%)*.!%%)777 7 s)7m $ 7 eC./ 0 7 3- 7"$7c]7"$7#7c]77777z5911 1 1 011 1111p(, $$$ $$e_ $  $$$$V59 DD DDH~D 01 D  (^ DDDDN- - - ----@! $"$" $" $"$"c] $"  $"$"$"$"V(,37"&*.%)*.!%%)!333 3 s)3m $ 3 eC./ 0 3 3- 3"$3c]3"$3#3c]33c]33333ld3iDIB(,37"&*.%)*.!%/// s)/m $/eC./ 0 / 3- / "$ /c]/"$/#/ ////dT#YC$tCy/((,! -- - s)-m $- -  - c] - (^----`-J--@S------r#