Crash with demo article #79

Open
opened 2026-02-08 13:07:23 +05:30 by halkeye · 3 comments
Contributor

Clicked the demo article, got 500

libmedium-55755bbdb6-cp59n libmedium  INFO  libmedium > libmedium: .
libmedium-55755bbdb6-cp59n libmedium For more information, see: https://git.batsense.net/realaravinth/libmedium
libmedium-55755bbdb6-cp59n libmedium Build info:
libmedium-55755bbdb6-cp59n libmedium Version: 0.1.0 commit: 02b20ba6774301b8241f5e9abbe6daa9662b527e
libmedium-55755bbdb6-cp59n libmedium Starting server on: http://0.0.0.0:7000
libmedium-55755bbdb6-cp59n libmedium  INFO  actix_server::builder > starting 4 workers
libmedium-55755bbdb6-cp59n libmedium  INFO  actix_server::server  > Actix runtime found; starting in Actix runtime
libmedium-55755bbdb6-cp59n libmedium  INFO  actix_web::middleware::logger > 10.244.3.88 "GET / HTTP/1.1" 200 1249 "-" "curl/8.14.1" 0.000569
libmedium-55755bbdb6-cp59n libmedium  INFO  actix_web::middleware::logger > 10.244.4.27 "GET / HTTP/1.1" 200 1249 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0" 0.000394
libmedium-55755bbdb6-cp59n libmedium thread 'actix-rt|system:0|arbiter:1' panicked at src/data.rs:182:22:
libmedium-55755bbdb6-cp59n libmedium called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }
libmedium-55755bbdb6-cp59n libmedium note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
libmedium-55755bbdb6-cp59n libmedium thread 'actix-rt|system:0|arbiter:2' panicked at src/data.rs:157:22:
libmedium-55755bbdb6-cp59n libmedium called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }

I would guess medium is blocking digitalocean ips, but i haven't looked into it at all.

Clicked the demo article, got 500 ``` libmedium-55755bbdb6-cp59n libmedium INFO libmedium > libmedium: . libmedium-55755bbdb6-cp59n libmedium For more information, see: https://git.batsense.net/realaravinth/libmedium libmedium-55755bbdb6-cp59n libmedium Build info: libmedium-55755bbdb6-cp59n libmedium Version: 0.1.0 commit: 02b20ba6774301b8241f5e9abbe6daa9662b527e libmedium-55755bbdb6-cp59n libmedium Starting server on: http://0.0.0.0:7000 libmedium-55755bbdb6-cp59n libmedium INFO actix_server::builder > starting 4 workers libmedium-55755bbdb6-cp59n libmedium INFO actix_server::server > Actix runtime found; starting in Actix runtime libmedium-55755bbdb6-cp59n libmedium INFO actix_web::middleware::logger > 10.244.3.88 "GET / HTTP/1.1" 200 1249 "-" "curl/8.14.1" 0.000569 libmedium-55755bbdb6-cp59n libmedium INFO actix_web::middleware::logger > 10.244.4.27 "GET / HTTP/1.1" 200 1249 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0" 0.000394 libmedium-55755bbdb6-cp59n libmedium thread 'actix-rt|system:0|arbiter:1' panicked at src/data.rs:182:22: libmedium-55755bbdb6-cp59n libmedium called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) } libmedium-55755bbdb6-cp59n libmedium note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace libmedium-55755bbdb6-cp59n libmedium thread 'actix-rt|system:0|arbiter:2' panicked at src/data.rs:157:22: libmedium-55755bbdb6-cp59n libmedium called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) } ``` I would guess medium is blocking digitalocean ips, but i haven't looked into it at all.

I've started getting the same error on my home IP I can (sometimes) view articles locally.

I think it's trying to pull GetPostLight in https://git.batsense.net/realaravinth/libmedium/src/branch/master/schemas/query.graphql:

query GetPostLight($id: ID!) {
    post(id: $id) {
        uniqueSlug
        creator {
            username
        }
    }
}

The closest query I could find in a successful article load (warning long):

query InteractivePostBodyQuery($postId: ID!, $showNotes: Boolean!, $includeHighlightResponseGroups: Boolean!) {
  post(id: $postId) {
    id
    highlights {
      ...InteractivePostBody_quote
      __typename
    }
    privateNotes @include(if: $showNotes) {
      ...PostBody_privateNote
      __typename
    }
    topHighlightResponseGroups @include(if: $includeHighlightResponseGroups) {
      ...InteractivePostBody_highlightGroup
      __typename
    }
    __typename
  }
}

fragment HighlightString_quote on Quote {
  id
  userId
  user {
    id
    name
    __typename
  }
  __typename
}

fragment SelectionMenuPopover_highlight on Quote {
  id
  userId
  paragraphs {
    id
    name
    __typename
  }
  startOffset
  endOffset
  ...HighlightString_quote
  __typename
}

fragment ActiveSelectionContext_highlight on Quote {
  ...SelectionMenuPopover_highlight
  __typename
  id
}

fragment HighlighSegmentContext_highlight on Quote {
  __typename
  endOffset
  id
  paragraphs {
    name
    __typename
  }
  startOffset
  userId
}

fragment NormalizeHighlights_paragraph on Paragraph {
  name
  text
  __typename
  id
}

fragment NormalizeHighlights_highlight on Quote {
  __typename
  endOffset
  startOffset
  paragraphs {
    ...NormalizeHighlights_paragraph
    __typename
  }
  id
}

fragment getParagraphHighlights_paragraph on Paragraph {
  name
  __typename
  id
}

fragment Markups_highlight on Quote {
  __typename
  id
  paragraphs {
    name
    __typename
  }
  startOffset
  endOffset
  userId
}

fragment TextParagraph_highlight on Quote {
  ...Markups_highlight
  __typename
  id
}

fragment ImageParagraph_highlight on Quote {
  id
  ...Markups_highlight
  __typename
}

fragment PostBodyParagraph_highlight on Quote {
  ...TextParagraph_highlight
  ...ImageParagraph_highlight
  __typename
  id
}

fragment getParagraphHighlights_highlight on Quote {
  __typename
  paragraphs {
    ...getParagraphHighlights_paragraph
    __typename
  }
  ...PostBodyParagraph_highlight
  id
}

fragment ParagraphStyleGrouping_highlight on Quote {
  ...PostBodyParagraph_highlight
  __typename
  id
}

fragment ParagraphLayoutGrouping_highlight on Quote {
  ...ParagraphStyleGrouping_highlight
  __typename
  id
}

fragment PostBodySection_highlight on Quote {
  ...ParagraphLayoutGrouping_highlight
  __typename
  id
}

fragment UserAvatar_user on User {
  id
  imageId
  name
  username
  __typename
}

fragment userUrl_user on User {
  __typename
  id
  customDomainState {
    live {
      domain
      __typename
    }
    __typename
  }
  hasSubdomain
  username
}

fragment UserAvatarLinkContainer_user on User {
  ...userUrl_user
  __typename
  id
}

fragment UserAvatarWithLink_user on User {
  ...UserAvatar_user
  ...UserAvatarLinkContainer_user
  __typename
  id
}

fragment PostNotesDetails_user on User {
  id
  imageId
  name
  username
  ...UserAvatarWithLink_user
  __typename
}

fragment PostNotesMarkers_highlight on Quote {
  __typename
  id
  endOffset
  paragraphs {
    id
    name
    __typename
  }
  startOffset
  user {
    id
    ...PostNotesDetails_user
    __typename
  }
  ...HighlightString_quote
}

fragment PostBody_highlight on Quote {
  paragraphs {
    id
    __typename
  }
  ...getParagraphHighlights_highlight
  ...PostBodySection_highlight
  ...PostNotesMarkers_highlight
  __typename
  id
}

fragment getParagraphPrivateNotes_privateNote on Note {
  id
  anchor
  author {
    id
    name
    username
    imageId
    mediumMemberAt
    membership {
      tier
      __typename
      id
    }
    ...userUrl_user
    __typename
  }
  content
  createdAt
  postId
  post {
    id
    creator {
      id
      __typename
    }
    collection {
      id
      __typename
    }
    __typename
  }
  replies {
    id
    author {
      id
      name
      username
      imageId
      mediumMemberAt
      membership {
        tier
        __typename
      }
      ...userUrl_user
      __typename
    }
    content
    createdAt
    noteId
    postId
    __typename
  }
  selectionStartOffset
  selectionEndOffset
  __typename
}

fragment normalizedBodyModel_privateNote on Note {
  ...getParagraphPrivateNotes_privateNote
  __typename
  id
}

fragment PostViewNoteCard_user on User {
  id
  name
  username
  ...UserAvatarWithLink_user
  ...userUrl_user
  __typename
}

fragment PostViewNoteCard_noteReply on NoteReply {
  id
  author {
    id
    ...PostViewNoteCard_user
    __typename
  }
  content
  createdAt
  noteId
  postId
  __typename
}

fragment PostViewNoteCard_privateNote on Note {
  id
  author {
    id
    ...PostViewNoteCard_user
    __typename
  }
  content
  createdAt
  postId
  post {
    id
    creator {
      id
      __typename
    }
    collection {
      id
      __typename
    }
    __typename
  }
  replies {
    id
    ...PostViewNoteCard_noteReply
    __typename
  }
  selectionStartOffset
  selectionEndOffset
  __typename
}

fragment PostAnnotationsMarker_privateNote on Note {
  ...PostViewNoteCard_privateNote
  __typename
  id
}

fragment TextParagraph_privateNote on Note {
  ...PostAnnotationsMarker_privateNote
  __typename
  id
}

fragment ImageParagraph_privateNote on Note {
  ...PostAnnotationsMarker_privateNote
  __typename
  id
}

fragment PostBodyParagraph_privateNote on Note {
  ...TextParagraph_privateNote
  ...ImageParagraph_privateNote
  __typename
  id
}

fragment ParagraphStyleGrouping_privateNote on Note {
  ...PostBodyParagraph_privateNote
  __typename
  id
}

fragment ParagraphLayoutGrouping_privateNote on Note {
  ...ParagraphStyleGrouping_privateNote
  __typename
  id
}

fragment PostBodySection_privateNote on Note {
  ...ParagraphLayoutGrouping_privateNote
  __typename
  id
}

fragment Markups_highlightGroup on HighlightGroup {
  __typename
  id
  startOffset
  endOffset
  paragraph {
    name
    __typename
    id
  }
}

fragment TextParagraph_highlightGroup on HighlightGroup {
  ...Markups_highlightGroup
  __typename
  id
}

fragment ImageParagraph_highlightGroup on HighlightGroup {
  id
  ...Markups_highlightGroup
  __typename
}

fragment PostBodyParagraph_highlightGroup on HighlightGroup {
  ...TextParagraph_highlightGroup
  ...ImageParagraph_highlightGroup
  __typename
  id
}

fragment ParagraphStyleGrouping_highlightGroup on HighlightGroup {
  ...PostBodyParagraph_highlightGroup
  __typename
  id
}

fragment ParagraphLayoutGrouping_highlightGroup on HighlightGroup {
  ...ParagraphStyleGrouping_highlightGroup
  __typename
  id
}

fragment PostBodySection_highlightGroup on HighlightGroup {
  ...ParagraphLayoutGrouping_highlightGroup
  __typename
  id
}

fragment buildQuoteParagraphForResponses_highlightGroup on HighlightGroup {
  __typename
  startOffset
  endOffset
  paragraph {
    text
    name
    id
    __typename
  }
  id
}

fragment InResponseToQuote_highlightGroup on HighlightGroup {
  ...buildQuoteParagraphForResponses_highlightGroup
  __typename
  id
}

fragment SimpleResponseEditor_highlightGroup on HighlightGroup {
  ...InResponseToQuote_highlightGroup
  __typename
  id
}

fragment PublishPostThreadedResponse_highlightGroup on HighlightGroup {
  id
  __typename
}

fragment ResponsesSidebarContext_highlightGroup on HighlightGroup {
  id
  ...SimpleResponseEditor_highlightGroup
  ...PublishPostThreadedResponse_highlightGroup
  __typename
}

fragment PostNotesResponsesButton_highlightGroup on HighlightGroup {
  responseCount
  ...ResponsesSidebarContext_highlightGroup
  __typename
  id
}

fragment PostNotesMarkers_highlightGroup on HighlightGroup {
  __typename
  id
  endOffset
  startOffset
  paragraph {
    id
    name
    __typename
  }
  ...PostNotesResponsesButton_highlightGroup
  ...ResponsesSidebarContext_highlightGroup
}

fragment PostBody_highlightGroup on HighlightGroup {
  ...PostBodySection_highlightGroup
  ...PostNotesMarkers_highlightGroup
  __typename
  id
}

fragment SelectionMenuPopover_highlightGroup on HighlightGroup {
  id
  paragraph {
    id
    name
    __typename
  }
  startOffset
  endOffset
  ...PostNotesResponsesButton_highlightGroup
  __typename
}

fragment ActiveSelectionContext_highlightGroup on HighlightGroup {
  ...SelectionMenuPopover_highlightGroup
  __typename
  id
}

fragment InteractivePostBody_quote on Quote {
  id
  ...ActiveSelectionContext_highlight
  ...HighlighSegmentContext_highlight
  ...NormalizeHighlights_highlight
  ...PostBody_highlight
  __typename
}

fragment PostBody_privateNote on Note {
  ...normalizedBodyModel_privateNote
  ...PostBodySection_privateNote
  __typename
  id
}

fragment InteractivePostBody_highlightGroup on HighlightGroup {
  ...PostBody_highlightGroup
  ...ActiveSelectionContext_highlightGroup
  __typename
  id
}

Wonder what's up with that.

Did they change their schema? Are they fingerprinting and rejecting queries? Is this rejecting clients who didn't pass a Cloudflare challenge at cdn-cgi/challenge-platform/h/g/precursor/...?

I've started getting the same error on my home IP I can (sometimes) view articles locally. I think it's trying to pull GetPostLight in https://git.batsense.net/realaravinth/libmedium/src/branch/master/schemas/query.graphql: ```graphql query GetPostLight($id: ID!) { post(id: $id) { uniqueSlug creator { username } } } ``` The closest query I could find in a successful article load (warning long): <details> ```graphql query InteractivePostBodyQuery($postId: ID!, $showNotes: Boolean!, $includeHighlightResponseGroups: Boolean!) { post(id: $postId) { id highlights { ...InteractivePostBody_quote __typename } privateNotes @include(if: $showNotes) { ...PostBody_privateNote __typename } topHighlightResponseGroups @include(if: $includeHighlightResponseGroups) { ...InteractivePostBody_highlightGroup __typename } __typename } } fragment HighlightString_quote on Quote { id userId user { id name __typename } __typename } fragment SelectionMenuPopover_highlight on Quote { id userId paragraphs { id name __typename } startOffset endOffset ...HighlightString_quote __typename } fragment ActiveSelectionContext_highlight on Quote { ...SelectionMenuPopover_highlight __typename id } fragment HighlighSegmentContext_highlight on Quote { __typename endOffset id paragraphs { name __typename } startOffset userId } fragment NormalizeHighlights_paragraph on Paragraph { name text __typename id } fragment NormalizeHighlights_highlight on Quote { __typename endOffset startOffset paragraphs { ...NormalizeHighlights_paragraph __typename } id } fragment getParagraphHighlights_paragraph on Paragraph { name __typename id } fragment Markups_highlight on Quote { __typename id paragraphs { name __typename } startOffset endOffset userId } fragment TextParagraph_highlight on Quote { ...Markups_highlight __typename id } fragment ImageParagraph_highlight on Quote { id ...Markups_highlight __typename } fragment PostBodyParagraph_highlight on Quote { ...TextParagraph_highlight ...ImageParagraph_highlight __typename id } fragment getParagraphHighlights_highlight on Quote { __typename paragraphs { ...getParagraphHighlights_paragraph __typename } ...PostBodyParagraph_highlight id } fragment ParagraphStyleGrouping_highlight on Quote { ...PostBodyParagraph_highlight __typename id } fragment ParagraphLayoutGrouping_highlight on Quote { ...ParagraphStyleGrouping_highlight __typename id } fragment PostBodySection_highlight on Quote { ...ParagraphLayoutGrouping_highlight __typename id } fragment UserAvatar_user on User { id imageId name username __typename } fragment userUrl_user on User { __typename id customDomainState { live { domain __typename } __typename } hasSubdomain username } fragment UserAvatarLinkContainer_user on User { ...userUrl_user __typename id } fragment UserAvatarWithLink_user on User { ...UserAvatar_user ...UserAvatarLinkContainer_user __typename id } fragment PostNotesDetails_user on User { id imageId name username ...UserAvatarWithLink_user __typename } fragment PostNotesMarkers_highlight on Quote { __typename id endOffset paragraphs { id name __typename } startOffset user { id ...PostNotesDetails_user __typename } ...HighlightString_quote } fragment PostBody_highlight on Quote { paragraphs { id __typename } ...getParagraphHighlights_highlight ...PostBodySection_highlight ...PostNotesMarkers_highlight __typename id } fragment getParagraphPrivateNotes_privateNote on Note { id anchor author { id name username imageId mediumMemberAt membership { tier __typename id } ...userUrl_user __typename } content createdAt postId post { id creator { id __typename } collection { id __typename } __typename } replies { id author { id name username imageId mediumMemberAt membership { tier __typename } ...userUrl_user __typename } content createdAt noteId postId __typename } selectionStartOffset selectionEndOffset __typename } fragment normalizedBodyModel_privateNote on Note { ...getParagraphPrivateNotes_privateNote __typename id } fragment PostViewNoteCard_user on User { id name username ...UserAvatarWithLink_user ...userUrl_user __typename } fragment PostViewNoteCard_noteReply on NoteReply { id author { id ...PostViewNoteCard_user __typename } content createdAt noteId postId __typename } fragment PostViewNoteCard_privateNote on Note { id author { id ...PostViewNoteCard_user __typename } content createdAt postId post { id creator { id __typename } collection { id __typename } __typename } replies { id ...PostViewNoteCard_noteReply __typename } selectionStartOffset selectionEndOffset __typename } fragment PostAnnotationsMarker_privateNote on Note { ...PostViewNoteCard_privateNote __typename id } fragment TextParagraph_privateNote on Note { ...PostAnnotationsMarker_privateNote __typename id } fragment ImageParagraph_privateNote on Note { ...PostAnnotationsMarker_privateNote __typename id } fragment PostBodyParagraph_privateNote on Note { ...TextParagraph_privateNote ...ImageParagraph_privateNote __typename id } fragment ParagraphStyleGrouping_privateNote on Note { ...PostBodyParagraph_privateNote __typename id } fragment ParagraphLayoutGrouping_privateNote on Note { ...ParagraphStyleGrouping_privateNote __typename id } fragment PostBodySection_privateNote on Note { ...ParagraphLayoutGrouping_privateNote __typename id } fragment Markups_highlightGroup on HighlightGroup { __typename id startOffset endOffset paragraph { name __typename id } } fragment TextParagraph_highlightGroup on HighlightGroup { ...Markups_highlightGroup __typename id } fragment ImageParagraph_highlightGroup on HighlightGroup { id ...Markups_highlightGroup __typename } fragment PostBodyParagraph_highlightGroup on HighlightGroup { ...TextParagraph_highlightGroup ...ImageParagraph_highlightGroup __typename id } fragment ParagraphStyleGrouping_highlightGroup on HighlightGroup { ...PostBodyParagraph_highlightGroup __typename id } fragment ParagraphLayoutGrouping_highlightGroup on HighlightGroup { ...ParagraphStyleGrouping_highlightGroup __typename id } fragment PostBodySection_highlightGroup on HighlightGroup { ...ParagraphLayoutGrouping_highlightGroup __typename id } fragment buildQuoteParagraphForResponses_highlightGroup on HighlightGroup { __typename startOffset endOffset paragraph { text name id __typename } id } fragment InResponseToQuote_highlightGroup on HighlightGroup { ...buildQuoteParagraphForResponses_highlightGroup __typename id } fragment SimpleResponseEditor_highlightGroup on HighlightGroup { ...InResponseToQuote_highlightGroup __typename id } fragment PublishPostThreadedResponse_highlightGroup on HighlightGroup { id __typename } fragment ResponsesSidebarContext_highlightGroup on HighlightGroup { id ...SimpleResponseEditor_highlightGroup ...PublishPostThreadedResponse_highlightGroup __typename } fragment PostNotesResponsesButton_highlightGroup on HighlightGroup { responseCount ...ResponsesSidebarContext_highlightGroup __typename id } fragment PostNotesMarkers_highlightGroup on HighlightGroup { __typename id endOffset startOffset paragraph { id name __typename } ...PostNotesResponsesButton_highlightGroup ...ResponsesSidebarContext_highlightGroup } fragment PostBody_highlightGroup on HighlightGroup { ...PostBodySection_highlightGroup ...PostNotesMarkers_highlightGroup __typename id } fragment SelectionMenuPopover_highlightGroup on HighlightGroup { id paragraph { id name __typename } startOffset endOffset ...PostNotesResponsesButton_highlightGroup __typename } fragment ActiveSelectionContext_highlightGroup on HighlightGroup { ...SelectionMenuPopover_highlightGroup __typename id } fragment InteractivePostBody_quote on Quote { id ...ActiveSelectionContext_highlight ...HighlighSegmentContext_highlight ...NormalizeHighlights_highlight ...PostBody_highlight __typename } fragment PostBody_privateNote on Note { ...normalizedBodyModel_privateNote ...PostBodySection_privateNote __typename id } fragment InteractivePostBody_highlightGroup on HighlightGroup { ...PostBody_highlightGroup ...ActiveSelectionContext_highlightGroup __typename id } ``` </details> Wonder what's up with that. Did they change their schema? Are they fingerprinting and rejecting queries? Is this rejecting clients who didn't pass a Cloudflare challenge at `cdn-cgi/challenge-platform/h/g/precursor/...`?

it's cloudflare.

            client: Client::builder()
                .user_agent("User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0")
                .connection_verbose(true)
                .build()
                .unwrap(),

err_safe.html (google says it's a WAF block?)

discussion on why at https://github.com/seanmonstar/reqwest/discussions/2227

Could you try enabling the http2 feature and also native-tls-alpn if you're using native-tls?

it seems http2 is no longer present (always on?) and native-tls-alpn was integrated into regular native-tls (https://docs.rs/reqwest/latest/reqwest/). enabling features = ["json", "native-tls"] did not fix the issue. Running samply record target/debug/libmedium shows stack traces referencing hyper. I'd say this means native-tls wasn't applied... except request's source code comments say default-tls "it currently uses native-tls"... idk

Additionally it seems reqwest issues a http/1.1 request?

I tried:

[dependencies.graphql_client]
features = ["reqwest"]
version = "0.16.0"

[dependencies.reqwest]
features = ["json", "native-tls-alpn"]
version = "0.12"

and we successfully fetch over http2, but still error out. sigh.

it's not the user agent; Bruno without user agent succeeds yet Reqwest with UA fails.

curl

curl 'https://medium.com/_/graphql' \
  --compressed \
  -X POST \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0' \
  -H 'Accept: */*' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Accept-Encoding: gzip, deflate, br, zstd' \
  -H 'content-type: application/json' \
  -H 'graphql-operation: VisitorQuery' \
  -H 'Origin: https://medium.com' \
  -H 'Connection: keep-alive' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Priority: u=4' \
  -H 'TE: trailers' \
  --data-raw '[{"operationName":"VisitorQuery","variables":{},"query":"query VisitorQuery {\n  visitor {\n    id\n    geolocation {\n      country\n      __typename\n    }\n    __typename\n  }\n}\n"}]'
  • removing User-Agent triggers the WAF error
  • removing graphql-operation triggers an interstitial (presumably thinks it's a HTTP request not graphql op?)

reqwest http1

[dependencies.reqwest]
features = ["json"]
version = "0.12"

actually logs website content! ...and fails anyway. despite the Cloudflare response having Content-Type: application/json; charset=utf-8 and a pile of data. i think it's a bug in our client somehow. Probably should debug it at some point...

EDIT: i'm past the blocking period, but it seems browser user-agent without browser headers gets nuked? but curl still requires the browser UA to not get nuked...

it's cloudflare. ```rust client: Client::builder() .user_agent("User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0") .connection_verbose(true) .build() .unwrap(), ``` [err_safe.html](/attachments/ee45e364-71af-4daa-8887-3860c77c291a) (google says it's a WAF block?) discussion on why at https://github.com/seanmonstar/reqwest/discussions/2227 > Could you try enabling the `http2` feature and also `native-tls-alpn` if you're using native-tls? it seems http2 is no longer present (always on?) and native-tls-alpn was integrated into regular native-tls (https://docs.rs/reqwest/latest/reqwest/). enabling `features = ["json", "native-tls"]` did not fix the issue. Running `samply record target/debug/libmedium` shows stack traces referencing hyper. I'd say this means native-tls wasn't applied... except request's source code comments say default-tls "it currently uses `native-tls`"... idk Additionally it seems reqwest issues a http/1.1 request? I tried: ```toml [dependencies.graphql_client] features = ["reqwest"] version = "0.16.0" [dependencies.reqwest] features = ["json", "native-tls-alpn"] version = "0.12" ``` and we successfully fetch over http2, but *still* error out. sigh. it's not the user agent; Bruno without user agent succeeds yet Reqwest with UA fails. ### curl ```sh curl 'https://medium.com/_/graphql' \ --compressed \ -X POST \ -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0' \ -H 'Accept: */*' \ -H 'Accept-Language: en-US,en;q=0.9' \ -H 'Accept-Encoding: gzip, deflate, br, zstd' \ -H 'content-type: application/json' \ -H 'graphql-operation: VisitorQuery' \ -H 'Origin: https://medium.com' \ -H 'Connection: keep-alive' \ -H 'Sec-Fetch-Dest: empty' \ -H 'Sec-Fetch-Mode: cors' \ -H 'Sec-Fetch-Site: same-origin' \ -H 'Priority: u=4' \ -H 'TE: trailers' \ --data-raw '[{"operationName":"VisitorQuery","variables":{},"query":"query VisitorQuery {\n visitor {\n id\n geolocation {\n country\n __typename\n }\n __typename\n }\n}\n"}]' ``` - removing User-Agent triggers the WAF error - removing graphql-operation triggers an interstitial (presumably thinks it's a HTTP request not graphql op?) ### reqwest http1 ```toml [dependencies.reqwest] features = ["json"] version = "0.12" ``` actually logs website content! ...and fails anyway. despite the Cloudflare response having `Content-Type: application/json; charset=utf-8` and a pile of data. i think it's a bug in our client somehow. Probably should debug it at some point... **EDIT:** i'm past the blocking period, but it seems browser user-agent without browser headers gets nuked? but curl still requires the browser UA to not get nuked...

I've assembled a working fork at https://git.batsense.net/nyanpasu64/libmedium/src/branch/fix (is your fingerprint SHA256:wUP77yJCCoTnbTu6LkYp2as3NzsZTIjfVWwllhcHkwY?).

I was able to successfully load /@wenting_zhang/getting-fluent-at-the-language-of-iconography-3415d03d0a8f. Though it does print a warning:

 INFO  libmedium::post       > Unknown type: SECTION_CAPTION

Strangely this caption appears in the JSON fetched by reqwest, and the DOM of the article at https://medium.com/@wenting_zhang/getting-fluent-at-the-language-of-iconography-3415d03d0a8f (window.__APOLLO_STATE__), but not in the actual text or alt-text visible in the browser. I decided to silence it, but don't know if this is the official website's behavior for all pages.

...looking at this page I think Scribe had better typography. you failed to add space between the blockquote and headings, since normally <p> handles margins but blockquotes lack them.

EDIT: turns out Scribe uses tufte-css:

nyanpasu64@servlet ~/code/scribe (main)> rg tufte  # (results reordered for clarity)
package.json
11:    "tufte-css": "^1.8.0",

yarn.lock
5541:tufte-css@^1.8.0:
5543:  resolved "https://registry.yarnpkg.com/tufte-css/-/tufte-css-1.8.0.tgz#16243dcde580716dbccaa40e097d1cbfbc7e8133"

src/css/app.css
1:@import "~tufte-css/tufte.css";
I've assembled a working fork at https://git.batsense.net/nyanpasu64/libmedium/src/branch/fix (is your fingerprint SHA256:wUP77yJCCoTnbTu6LkYp2as3NzsZTIjfVWwllhcHkwY?). I was able to successfully load /@wenting_zhang/getting-fluent-at-the-language-of-iconography-3415d03d0a8f. Though it does print a warning: ``` INFO libmedium::post > Unknown type: SECTION_CAPTION ``` Strangely this caption appears in the JSON fetched by reqwest, and the DOM of the article at https://medium.com/@wenting_zhang/getting-fluent-at-the-language-of-iconography-3415d03d0a8f (`window.__APOLLO_STATE__`), but not in the actual text or alt-text visible in the browser. I decided to silence it, but don't know if this is the official website's behavior for all pages. ...looking at this page I think Scribe had better typography. you failed to add space between the blockquote and headings, since normally `<p>` handles margins but blockquotes lack them. **EDIT:** turns out Scribe uses tufte-css: ``` nyanpasu64@servlet ~/code/scribe (main)> rg tufte # (results reordered for clarity) package.json 11: "tufte-css": "^1.8.0", yarn.lock 5541:tufte-css@^1.8.0: 5543: resolved "https://registry.yarnpkg.com/tufte-css/-/tufte-css-1.8.0.tgz#16243dcde580716dbccaa40e097d1cbfbc7e8133" src/css/app.css 1:@import "~tufte-css/tufte.css"; ```
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
realaravinth/libmedium#79
No description provided.