i2TdZddlZddlmZmZmZmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZmZmZmZddlmZded dfd Zded efd Zd eeed ee fd Zdeeefd eeeffdZ deded e fdZ! d%dddededeedee"d e f dZ#dddededee"d e fdZ$ d%dddededee deedee"d ee f dZ%deded e&fdZ' d&dddedede(dee(dee"d e f dZ) d&dddedede(dee(dee"d e f dZ*dede d efd Z+d!ed"ed efd#Z,ded efd$Z-dS)'z. Crawling functionality for Firecrawl v2 API. N)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsResponseCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_inputrequestreturnc|jr|jstd|j|jdkrtd|jt |jdSdS)z Validate crawl request parameters. Args: request: CrawlRequest to validate Raises: ValueError: If request is invalid URL cannot be emptyNrzLimit must be positive)urlstrip ValueErrorlimitscrape_optionsr)rs d/home/agentuser/.hermes/hermes-agent/venv/lib/python3.11/site-packages/firecrawl/v2/methods/crawl.py_validate_crawl_requestr#s ;0gk//110.///} W]a%7%71222) 677777*)ct|d|ji}|jr |j|d<|jt |j}|r||d<|dd}|dd|dd|dd|jCt|jtr |j|d<n|jd |d<d d d d dddddddddddd}| D]!\}}||vr||||<"| |d|vr8t|dtr|d |d<|S)z Prepare crawl request for API submission. Args: request: CrawlRequest to prepare Returns: Dictionary ready for API submission rpromptN scrapeOptionsT) exclude_none exclude_unsetr!webhook)r( includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsignoreRobotsTxtrobotsUserAgentdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr.ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsignore_robots_txtrobots_user_agentr6max_concurrencyregex_on_full_urlzero_data_retention integration) r#rr&r!r model_dumppopr* isinstancestritemsupdater)rdata scrape_data request_datafield_mappings snake_case camel_cases r"_prepare_crawl_requestrT&sG$$$ 7; D~( X),W-CDD  0$/D !%%4t%LLLUD!!!Xt$$$%t,,," gos + + L%oDOO&o88d8KKDO('2#:$<2 4-..+-2N&#1"6"6"8"8<< J  % %+// ;;D  KK D,?!E!E"=17799] Kr$ data_listc g}|pgD]C}t|tr,|tdit |D|S)N)rJdictappendr r)rU documentsdoc_datas r"_parse_crawl_documentsr\qsa "IOMM h % % M   XKK(@(J(JKK L L L r$ response_datac |ds#t|dd|d|dd|dd|dd|d |d t|d gd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrNrbrcrd credits_used expires_atrgrN)get Exceptionr\)r]s r"_parse_crawl_status_responsermys   Y ' 'N ))'3KLLMMM ##H--"&&{A66""7A..%))-;;#'' 44!!&))&}'8'8'D'DEE  r$clientct|}|d|}|jst|d|}|dr7|d|dd}t d i|St|dd) aP Start a crawl job for a website. Args: client: HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlz start crawlr_idr)rprr`raNrW)rTpostokrjsonrkr rl)rnrrPresponser]job_datas r" start_crawlrvs*'22L{{; 55H ;7h 666MMOOM##N##D)) $$U++   ((x((( ))'3KLLMMMr$request_timeoutjob_idpagination_configrxc |d||}|jst|d|}t |}|d}|r|jnd}|rC|dr;|r|jt||jkst||d|||}t|d |d |d |d |d |s|dnd|S)aK Get the status of a crawl job. Args: client: HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination behavior request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with current status and data Raises: Exception: If the status check fails /v2/crawl/timeoutzget crawl statusrNTrgNrwrbrcrdrirjrh) rkrrrrsrm auto_paginate max_resultslen_fetch_all_pagesr ) rnryrzrxrtr]payloadrZrs r"get_crawl_statusrs.2zz/v//zIIH ;<h(:;;;MMOOM*=99GI8IR%33dM       ) 5  NN/; ; ;$  FO  +     x +&g^,<($1 ;WV__t    r$next_urlc $|||}|jst|d|}t |}t |d|d|d|d|d|d|d  S) a Fetch a single page of crawl results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response r}zget crawl status pagerbrcrdrirjrgrNrh)rkrrrrsrmr )rnrrxrtr]rs r"get_crawl_status_pagers(zz(Oz<'>   %DN,<,cN d d d MMOO  7 BBLL    E %V, ' 'H&3y>>[+H+H   X & & & &  "s9~~'D'D #6* a M &P s"C22 C?>C?c|d|}|jst|d|}|ddkS)a Cancel a running crawl job. Args: client: HTTP client instance job_id: ID of the crawl job to cancel Returns: bool: True if the crawl was cancelled, False otherwise Raises: Exception: If the cancellation fails r|z cancel crawlrb cancelled)deleterrrrsrk)rnryrtr]s r" cancel_crawlr^s^}}2&2233H ;8h777MMOOM   X & &+ 55r$ poll_intervalr~ctj} t|||}|jdvr|S|0tj|z |krt d|d|dtj|d)a Wait for a crawl job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the crawl job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) request_timeout: Optional timeout (in seconds) for each status request Returns: CrawlJob when job completes Raises: Exception: If the job fails TimeoutError: If timeout is reached Trw)rcfailedrNz Crawl job z did not complete within z seconds)rrrrb TimeoutErrorsleep)rnryrr~rxr crawl_jobs r"wait_for_crawl_completionrus2!!J"$  +     C C C   DN$4$4z$AW#L#L^F^^W^^^__ _ =!!!!"r$cdt||}|j}||n|}t|||||S)a Start a crawl job and wait for it to complete. Args: client: HTTP client instance request: CrawlRequest containing URL and options poll_interval: Seconds between status checks timeout: Maximum seconds to wait for the entire crawl job to complete (None for no timeout) request_timeout: Timeout (in seconds) for each individual HTTP request, including pagination requests when fetching results. If there are multiple pages, each page request gets this timeout Returns: CrawlJob when job completes Raises: ValueError: If request is invalid Exception: If the crawl fails to start or complete TimeoutError: If timeout is reached Nrw)rvrpr)rnrrr~rxrryeffective_request_timeouts r"crawlrsU8FG,,I \F4C3NT[ %1    r$c|jr|jstd|jr|jstd|j|jd}|d|}|jst |d|}|dr{|di}i}dd d d d d ddddddddd}d|vr2|d}t|trtd(i||d<n||d<| D]\} } | |vr| dkr|| || } i} ddddddd } | D]\}}|| vr | || |<d!| vr8| d!}t|trd"d#lm}||$| d!<n|| d!<| D]\}}|| vr |d!kr|| |<| || <|| || <| D]\}}||vr|||<d%|vr |d%|d%<t!d(i|St#|d&d'))al Get crawl parameters from LLM based on URL and prompt. Args: client: HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing suggested crawl options Raises: ValueError: If request is invalid Exception: If the operation fails rzPrompt cannot be empty)rr&z/v2/crawl/params-previewzcrawl params previewr_rNr:r;r<r.r=r>r?r@rArBrCrDr!rF)r+r,r-r.r/r0r1r2r3r4r5r7r'r9r*r'N include_tags exclude_tagsonly_main_contentwait_forskip_tls_verificationremove_base64_images) includeTags excludeTagsonlyMainContentwaitForskipTlsVerificationremoveBase64Imagesformatsr) ScrapeFormats)rrr`rarW)rrrr&rqrrrrsrkrJrXrrLlisttypesrrrl)rnrrPrtr] params_dataconverted_paramsrQ webhook_datarSrRscrape_opts_dataconverted_scrape_optsscrape_field_mappings scrape_camel scrape_snake formats_datarkeyvalues r"crawl_params_previewrsz ;0gk//110./// >3!5!5!7!731222{.L {{5|DDH ;@h(>???MMOOM##ON#''33 ++!6 %>&@!6"8122/-!6  $  # #&y1L,-- ;.;.K.Kl.K.K ++.: +&4&:&:&<&<$ K$ K "J [((00[5L5X'2:'>$,.)'5'5+>#-/F.D --)7L6Q6Q6S6Saa2 l'+;;;BRS_B`1,?!$444'7 'B %lD99L======?L}Ua?b?b?b1)<&>?? U&;;;y@P@P9>1#63H$Z003>z3J$Z0&++-- . .JC.(((- %  % %*7 *B Y '22!1222 ))'3KLLMMMr$ http_clientcrawl_idc|d|d}|jst|d |}|d|}|dg|d|dgd}t d i|S#t $r}t d |d }~wwxYw) a- Get errors from a crawl job. Args: http_client: HTTP client for making requests crawl_id: The ID of the crawl job Returns: CrawlErrorsResponse containing errors and robots blocked URLs Raises: Exception: If the request fails r|z/errorszcheck crawl errorsrNerrors robotsBlockedrobots_blocked)rrz'Failed to parse crawl errors response: NrW)rkrrrrsrrl)rrrtbodyr normalizedes r"get_crawl_errorsrGs=H===>>H ;>h(<=== G}}((64((kk(B//%kk/7;;GWY[;\;\]]  #00Z000 GGGE!EEFFFGsA6B)) C 3CC c |d}|jst|d|}|ds#t |dd|dg}g}|D]}t |t rz||d|d|d |d |d d td d|DS)z Get a list of currently active crawl jobs. Args: client: HTTP client instance Returns: ActiveCrawlsResponse containing a list of active crawl jobs Raises: Exception: If the request fails z/v2/crawl/activezget active crawlsr_r`racrawlsrpteamIdteam_idroptions)rprrrTc&g|]}tdi|S)rW)r).0ncs r" z%get_active_crawls..s&5d5d5dBk6G6GB6G6G5d5d5dr$)r_r) rkrrrrsrlrJrXrYr)rnrtr crawls_innormalized_crawlscs r"get_active_crawlsrgs8zz,--H ;=h(;<<< ==??D 88I  E*BCCDDD2&&I  a     $ $eeDkk55155+;+;<<uuU||55++ &&    5d5dRc5d5d5d e e eer$)N)rN).__doc__rtypingrrrrrrr r r r r rrrrrrutilsrrrrutils.normalizerr#rXrTr\rKrmrvfloatrrrboolrintrrrrrrWr$r"rs ,,,,,,,,,,,, gfffffffffff6666668\8d8888*HLHTHHHHVhtCy&9d8n S#X 4S>     N N\ Nm N N N NL59> (, >>> > > 01> e_ >  >>>>J(, $$$ $$e_ $  $$$$V59 H (, HHH HHH~H 01 H e_ H (^HHHHV66S6T66664! +" (, +"+"+" +" +"+"c] +" e_ +"+"+"+"+"b! ) (, ))) ) ))c] ) e_ )))))XuNuN6HuN_uNuNuNuNpG*GG@SGGGG@ fj f-A f f f f f fr$