i7 ddlmZmZmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZddlZd ed efd Zd eeed ee fd Zdeeefd eeeffdZded ed efdZ d dddededeedee d ef dZ!dddededee d efdZ" d dddededee deedee d ee f dZ#deded e$fdZ%ded e d e fdZ&deded e fdZ'ded efdZ(dS)!)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig)handle_response_error)prepare_scrape_options)AsyncHttpClient)normalize_document_inputNrequestreturncb|jr|jstdd|ji}|jr |j|d<|jt |j}|r||d<|jCt|jtr |j|d<n|j d|d<| dd}| dd| dd| d dd d d d dddddddddddd}| D]!\}}||vr| |||<"| |t|dd2tt|d|d<|S)NURL cannot be emptyurlprompt scrapeOptionswebhookT) exclude_none)r exclude_unsetscrape_options includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsignoreRobotsTxtrobotsUserAgentdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr%ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsignore_robots_txtrobots_user_agentr-max_concurrencyregex_on_full_urlzero_data_retention integration)rstrip ValueErrorrr!rr isinstancestr model_dumppopitemsupdategetattr)rdataopts request_datafield_mappingssnakecamels h/home/agentuser/.hermes/hermes-agent/venv/lib/python3.11/site-packages/firecrawl/v2/methods/aio/crawl.py_prepare_crawl_requestrOs ;0gk//110./// 7; D~( X)%g&<==  )$(D !" gos + + L%oDOO%o88d8KKDO%%4t%LLLUD!!!Xt$$$%t,,,''2#:$<2 4-..+-2N"',,..22 u L &**511DKKK w t,,8!''="A"ABBHHJJ] K data_listcg}|pgD]E}t|tr.t|}|t di|F|S)N)rAdictrappendr )rQ documentsdoc_data normalizeds rN_parse_crawl_documentsrYDsb "IO55 h % % 51(;;J   X33 33 4 4 4 rPbodyc |ds#t|dd|d|dd|dd|dd|d |d t|d gd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrHr_r`ra credits_used expires_atrdrH)get ExceptionrY)rZs rN_parse_crawl_status_responserjMs 88I  E*BCCDDD((8$$XXk1--'1%%22hh{++  &txx';';<<  rPclientcKt|}|d|d{V}|jdkrt|d|}|dr7t |d|dSt|d d ) aV Start a crawl job for a website. Args: client: Async HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlNz start crawlr\idr)rnrr]r^)rOpost status_coderjsonrhr ri)rkrpayloadresponserZs rN start_crawlrt\s%W--G[[g66666666Hs""h 666 ==??D xx EDHHUOODDDD DHHW&>?? @ @@rPrequest_timeoutjob_idpagination_configrvc K|d||d{V}|jdkrt|d|}t |}|d}|r|jnd}|r(|dr t ||d||| d{V}t|d |d |d |d |d|s|dnd|S)a[ Get the status of a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination limits request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with job information Raises: Exception: If the status check fails /v2/crawl/timeoutNrmzget crawl statusrHTrdrur_r`rarfrgre)rhrprrqrj auto_paginate_fetch_all_pages_asyncr ) rkrwrxrvrsrZrrrVr}s rNget_crawl_statusrus<0ZZ 5V 5 5ZOOOOOOOOHs""h(:;;; ==??D*400GI8IR%33dM  0  FO  +           x +&g^,<($1 ;WV__t    rPnext_urlc <K|||d{V}|jdkrt|d|}t |}t |d|d|d|d|d |d |d  S) a Fetch a single page of crawl results using the provided next URL. Args: client: Async HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response r{Nrmzget crawl status pager_r`rarfrgrdrHre)rhrprrqrjr )rkrrvrsrZrrs rNget_crawl_status_pagers(ZZ/ZBBBBBBBBHs""h(?@@@ ==??D*400G x +&g^,<( V_ V_   rPinitial_documentscK|}|}d}|r|jnd}|r|jnd} |r|jnd} t j} |r|||krn| t j| z | krn|||d{V} | jdkr8ddl} | d}| dd| jin| } t|}n#t$rYnawxYw|d D].}| t|| krn||/| t|| krn|d }|d z }||S) a Fetch all pages of crawl results asynchronously. Args: client: Async HTTP client instance next_url: URL for the next page initial_documents: Documents from the first page pagination_config: Optional configuration for pagination limits request_timeout: Optional timeout (in seconds) for the underlying HTTP request Returns: List of all documents from all pages rNr{rm firecrawlzFailed to fetch next pagerp)extrarHrd)copy max_pages max_results max_wait_timetime monotonicrhrplogging getLoggerwarningrqrjrilenrU)rkrrrxrvrV current_url page_countrrr start_timersrlogger page_data page_payloaddocuments rNr~r~s*"&&((IKJ0AJ!++dI3DN#//$K7HR%33dM!!J %  !zY'>'>   %DN,<,cN d d d MMOO  7 BBLL    E %V, ' 'H'c)nn .K.K   X & & & &  ##i..K*G*G #6* a K %N s.C>> D  D cK|d|d{V}|jdkrt|d|}|ddkS)a  Cancel a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job Returns: True if cancellation was successful Raises: Exception: If the cancellation operation fails rzNrmz cancel crawlr_ cancelled)deleterprrqrh)rkrwrsrZs rN cancel_crawlrsv]]#8#8#899999999Hs""h777 ==??D 88H   ,,rPcK|jr|jstd|jr|jstd|j|jd}|d|d{V}|jdkrt |d|}|ds#t|d d |d i}i}d d ddddddddddddd}| D]\}} ||vr |||| <d|vr |d} | |d<d|vr |d|d<tdi|S)au Preview crawl parameters before starting a crawl job. Args: client: Async HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing crawl configuration Raises: ValueError: If request is invalid Exception: If the parameter preview fails rzPrompt cannot be empty)rrz/v2/crawl/params-previewNrmzcrawl params previewr\r]r^rHr1r2r3r%r4r5r6r7r8r9r:r;r!r=)r"r#r$r%r&r'r(r)r*r+r,r.rr0rrrS) rr?r@rrorprrqrhrirEr ) rkrrrrsrZ params_data convertedmappingrMrLwks rNcrawl_params_previewr.s ;0gk//110./// >3!5!5!7!731222kW^<??? ==??D 88I  E*BCCDDD((62&&K "I''2!:"<24-..+)2G  22 u K  *51Ie K  #! )D#I )  ' 'Y ' ''rPcrawl_idchK|d|dd{V}|jdkrt|d|}|d|}|dg|d|d gd }t d i|S) a" Get errors from a crawl job. Args: client: Async HTTP client instance crawl_id: ID of the crawl job Returns: CrawlErrorsResponse with errors and robots blocked Raises: Exception: If the error check operation fails rzz/errorsNrmzcheck crawl errorsrHerrors robotsBlockedrobots_blocked)rrrS)rhrprrqr)rkrrsrZrrrXs rNget_crawl_errorsresZZ >X > > >????????Hs""h(<=== ==??Dhhvt$$G++h++!++ow{{CSUW7X7XYYJ  , , , ,,rPc K|dd{V}|jdkrt|d|}|ds#t |dd|dg}g}|D]}t |t rz||d |d |d |d |d dtdd|DS)z Get active crawl jobs. Args: client: Async HTTP client instance Returns: ActiveCrawlsResponse with active crawl jobs Raises: Exception: If the active crawl jobs operation fails z/v2/crawl/activeNrmzget active crawlsr\r]r^crawlsrnteamIdteam_idroptions)rnrrrTc&g|]}tdi|S)rS)r).0ncs rN z%get_active_crawls..s&5]5]5]Bk6G6GB6G6G5]5]5]rP)r\r) rhrprrqrirArTrUr)rkrsrZ crawls_inrXcs rNget_active_crawlsrsQZZ 233333333Hs""h(;<<< ==??D 88I  E*BCCDDD2&&IJ  a      eeDkk55155+;+;<<uuU||55++     5]5]R\5]5]5] ^ ^ ^^rP)N))typingrrrrtypesrr r r r r rrrrrutils.error_handlerrutils.validationrutils.http_client_asyncrutils.normalizerrrTrOrYrBrjrtfloatrrr~boolrrrrrSrPrNrs,,,,,,,,,,,,                          988888666666666666777777 +L+T++++\htCy&9d8n tCH~ $sCx.    AoA AAAAA8593 (, 333 3 3 013 e_ 3  3333t(, !!! !!e_ !  !!!!P59 G (, GGG GGH~G 01 G e_ G (^GGGGT-------*4(4(AS4(Xg4(4(4(4(n-?-c-FY----4_O_8L______rP