packages feed

shodan 0.3.0.0 → 0.4.0.0

raw patch · 10 files changed

+569/−22 lines, 10 filesnew-component:exe:shodan-domainPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

+ Shodan: DnsRecord :: Text -> Text -> Text -> Text -> DnsRecord
+ Shodan: DomainInfo :: Text -> [Text] -> [DnsRecord] -> [Text] -> Bool -> DomainInfo
+ Shodan: InvalidDomain :: ShodanError
+ Shodan: [dnsLastSeen] :: DnsRecord -> Text
+ Shodan: [dnsSubdomain] :: DnsRecord -> Text
+ Shodan: [dnsType] :: DnsRecord -> Text
+ Shodan: [dnsValue] :: DnsRecord -> Text
+ Shodan: [domainMore] :: DomainInfo -> Bool
+ Shodan: [domainName] :: DomainInfo -> Text
+ Shodan: [domainRecords] :: DomainInfo -> [DnsRecord]
+ Shodan: [domainSubdomains] :: DomainInfo -> [Text]
+ Shodan: [domainTags] :: DomainInfo -> [Text]
+ Shodan: [matchAlpn] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchCertificateNames] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchDomains] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchHostnames] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchTimestamp] :: SearchMatch -> Maybe Text
+ Shodan: data DnsRecord
+ Shodan: data DomainInfo
+ Shodan: getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.DnsRecord
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.DomainInfo
+ Shodan: instance GHC.Classes.Eq Shodan.DnsRecord
+ Shodan: instance GHC.Classes.Eq Shodan.DomainInfo
+ Shodan: instance GHC.Show.Show Shodan.DnsRecord
+ Shodan: instance GHC.Show.Show Shodan.DomainInfo
- Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> Maybe Text -> SearchMatch
+ Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> Maybe Text -> Maybe Text -> Maybe [Text] -> Maybe [Text] -> Maybe [Text] -> Maybe [Text] -> SearchMatch

Files

CHANGELOG.md view
@@ -2,6 +2,37 @@  This package follows the [Haskell Package Versioning Policy](https://pvp.haskell.org/). +## 0.4.0.0 — 2026-11-21++### Breaking changes++- Add `InvalidDomain` to `ShodanError`; exhaustive handlers need a new case.+- Append timestamp, hostname, domain, ALPN, and certificate DNS name fields to the public `SearchMatch`+  constructor. Positional construction and patterns need five additional fields;+  this datatype change requires a PVP major bump.++### Additions++- Add `getDomainInfo`, `DomainInfo`, and `DnsRecord` for a single page of domain+  DNS information, with explicit page selection and validation before HTTP.+- Add `shodan-domain DOMAIN [PAGE]`, including subdomains, records, last-seen+  timestamps, and the next-page flag. Each lookup uses one query credit;+  history and record-type filters are not implemented yet.+- Add domain parsing, HTTP, and argument validation tests and documentation.+  Verify a live domain lookup manually; use `domainMore` rather than assuming+  a fixed number of records per page.+- Add `matchTimestamp` (UTC collection time preserved as text), `matchHostnames`,+  `matchDomains`, and nested `ssl.alpn` as `matchAlpn`.+- Missing or null metadata becomes `Nothing`; empty lists remain `Just []`.+  Incorrect JSON types are rejected. Timestamp syntax is not validated.+- Show collection time, hostnames, domains, and ALPN in the search executable.+- Add `matchCertificateNames` and a `CERTIFICATE_NAMES` column for DNS names+  from the leaf certificate SAN, including wildcards. Decode DER and escaped+  byte representations; omit non-DNS entries and reject malformed SAN data.+  Certificate names are candidates, not verified DNS mappings or live sites.+- Add local parsing tests and document the meaning and limits of the metadata.+- Verify search metadata manually; automated tests remain local.+ ## 0.3.0.0 — 2026-11-20  ### Breaking changes
README.md view
@@ -4,12 +4,13 @@  Licensed under the [BSD 3-Clause License](LICENSE). -Current version: **0.3.0.0**. See the [changelog](CHANGELOG.md)+Current version: **0.4.0.0**. See the [changelog](CHANGELOG.md) for additions and the `SearchMatch` constructor change requiring a PVP major bump.  This project supports `GET /api-info` (`ApiInfo` and `UsageLimits`), `GET /shodan/host/count` (`CountResult`), and one-page-`GET /shodan/host/search` (`SearchResult`). The API-info example fields and+`GET /shodan/host/search` (`SearchResult`), plus one-page+`GET /dns/domain/{domain}` (`DomainInfo`). The API-info example fields and the count response's `total` field are required; unknown fields are ignored. Numeric values use `Integer`; count totals must be nonnegative. @@ -152,10 +153,91 @@ required; the total is a nonnegative `Integer`. Facets follow the same parsing rules as `CountResult`. +Search metadata is also available through `matchTimestamp :: Maybe Text`,+`matchHostnames :: Maybe [Text]`, `matchDomains :: Maybe [Text]`, and+`matchAlpn :: Maybe [Text]`. The timestamp is the UTC banner collection time,+preserved as text without date validation. Hostnames are based on reverse DNS;+these lists do not enumerate all websites hosted on an IP.++`matchAlpn` reads the nested `ssl.alpn` list (for example `h2`, `http/1.1`).+It records Shodan's observation, not the protocol used for our API request.+Missing ALPN data does not establish that HTTP/2 or HTTP/3 is unsupported.+See the [banner specification](https://datapedia.shodan.io/) and+[TLS fields](https://datapedia.shodan.io/property/ssl.html).++All four metadata fields allow absent or null values as `Nothing`. Missing or+null `ssl` also produces `Nothing` for ALPN. A present non-null `ssl` must be an+object; non-text timestamps and lists containing non-text elements are rejected.+List order is preserved, and `Just []` is distinct from missing data.+The CLI adds `COLLECTED_UTC`, `HOSTNAMES`, `DOMAINS`, and `ALPN` columns.+Lists use commas, empty lists display `[]`, and missing values display `-`.++`matchCertificateNames :: Maybe [Text]` extracts DNS names from the leaf+certificate's `ssl.cert.extensions` entry named `subjectAltName` (SAN).+The search example displays them in `CERTIFICATE_NAMES`. Wildcards such as+`*.example.com` and list order are preserved; IP, URI, and email SAN entries+are omitted. These are certificate name candidates, not verified DNS mappings,+live sites, or a complete subdomain inventory. No DNS lookup or extra API call+is made, and the certificate common name is not used as a fallback.++Missing/null certificate or extensions, or no SAN extension, yields `Nothing`+(displayed as `-`). A present SAN containing no DNS names yields `Just []`+(displayed as `[]`). The parser handles DER bytes represented as JSON text and+literal Python-style byte escapes, as in the official Shodan example. Invalid+extension structure, non-text SAN data, and malformed DER reject the response.+Other extension contents are ignored. This reads names only; it does not validate+certificate signatures, expiry, or domain-name syntax.+ Optional facet requests and field selection are not supported yet. Search tests use local JSON fixtures and a mock HTTP server, and do not consume Shodan query credits. A one-page live search has also succeeded manually. +## Domain DNS information++Retrieve one page of Shodan's observed subdomains and DNS records:++```haskell+getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)+```++With `SHODAN_API_KEY` set, run:++```sh+cabal run shodan-domain -- example.com+cabal run shodan-domain -- example.com 2+```++The optional page defaults to 1. Shodan documents pages of 100 DNS records,+but live responses can contain more; use `domainMore`, not the record count,+to decide whether another page is available.+According to the [official DNS API documentation](https://developer.shodan.io/api),+each lookup uses one query credit. No extra pages or retries are automatic.+Check `domainMore` before requesting the next page. This endpoint queries+Shodan's stored DNS information, not a live DNS lookup, and cannot guarantee+that every subdomain is known or that a mapping is still current.++`DomainInfo` contains `domainName`, `domainTags`, `domainRecords`,+`domainSubdomains` (relative names as returned), and `domainMore`.+Each `DnsRecord` contains `dnsSubdomain`, `dnsType`, `dnsValue`, and+`dnsLastSeen`. All response fields are required; missing/null fields and wrong+JSON types are rejected, while unknown fields are ignored. List order and+wildcards are preserved. DNS types, values, and timestamps remain text without+syntax validation or IP-only assumptions. An empty record subdomain means the+apex domain. The CLI prints fully qualified record names, types, values, and+last-seen timestamps, plus relative subdomains and the `More` flag.++Supply an ASCII domain name (punycode for international names), without a URL+or trailing dot. Labels must have 1–63 letters, digits, or hyphens, without a+leading/trailing hyphen; the whole name must be at most 253 characters.+Invalid names return `Left InvalidDomain` and pages below 1 return+`Left InvalidPage`, before any HTTP request. The API key is handled through+the same client and redacted errors as the other endpoints.++This first implementation uses the API defaults: no history and no DNS+record-type filter. `/dns/resolve` and `/dns/reverse` are not implemented.+Automated DNS parsing and HTTP tests use fixtures and a local server; no+query credits are needed. A live domain lookup has also succeeded manually.+ ## Build  The initial development compiler is GHC 9.8.4.@@ -186,7 +268,7 @@ ```sh stylish-haskell -i src/Shodan.hs test/*.hs examples/api-info/Main.hs stylish-haskell -i examples/count/Main.hs-stylish-haskell -i examples/search/Main.hs+stylish-haskell -i examples/search/Main.hs examples/domain/Main.hs cabal test ``` @@ -200,6 +282,8 @@ - `examples/count/Main.hs`: count-query executable using the same environment variable. - `examples/search/Main.hs`: one-page search executable with an optional page argument. - `test/Main.hs`: response parsing and HTTP tests against a local mock server.+- `examples/domain/Main.hs`: one-page domain DNS lookup executable.+- `test/DnsTests.hs`: domain and DNS record parsing tests. - `test/SearchTests.hs`: search-response JSON parsing tests.  No API key is needed to build or test. Automated tests do not call Shodan.
examples/api-info/Main.hs view
@@ -28,4 +28,5 @@ errorMessage NetworkError = "Could not reach Shodan (connection failure or timeout)." errorMessage (HttpError status) = "Shodan returned HTTP " ++ show status ++ "." errorMessage InvalidResponse = "Shodan returned an unexpected response."-errorMessage InvalidPage = "Search page numbers must be at least 1."+errorMessage InvalidPage = "Page numbers must be at least 1."+errorMessage InvalidDomain = "Expected an ASCII DNS domain name without a URL or trailing dot."
+ examples/domain/Main.hs view
@@ -0,0 +1,55 @@+module Main (main) where++import           Data.Char          (isSpace)+import           Data.List          (intercalate)+import qualified Data.Text          as Text+import           Shodan             (DnsRecord (..), DomainInfo (..),+                                     getDomainInfo, newClient)+import           System.Environment (getArgs, lookupEnv)+import           System.Exit        (die)+import           Text.Read          (readMaybe)++main :: IO ()+main = do+  arguments <- getArgs+  (domain, page) <- parseArguments arguments+  configuredKey <- lookupEnv "SHODAN_API_KEY"+  case configuredKey of+    Nothing -> die "Set the SHODAN_API_KEY environment variable before running."+    Just key+      | all isSpace key -> die "SHODAN_API_KEY must not be empty or whitespace."+      | otherwise -> do+          client <- newClient (Text.pack key)+          result <- getDomainInfo client domain page+          case result of+            Left err -> die ("Shodan domain lookup failed: " ++ show err)+            Right found -> do+              putStrLn ("Domain: " ++ Text.unpack (domainName found))+              putStrLn ("Page: " ++ show page)+              putStrLn ("More: " ++ show (domainMore found))+              putStrLn ("Subdomains: " ++ renderNames (domainSubdomains found))+              putStrLn "HOSTNAME\tTYPE\tVALUE\tLAST_SEEN"+              mapM_ (printRecord (domainName found)) (domainRecords found)++parseArguments :: [String] -> IO (Text.Text, Int)+parseArguments [domain] = parseArguments [domain, "1"]+parseArguments [domain, pageText]+  | not (all isSpace domain)+  , Just page <- (readMaybe pageText :: Maybe Integer)+  , page >= 1+  , page <= toInteger (maxBound :: Int) = pure (Text.pack domain, fromInteger page)+parseArguments _ = die "Usage: shodan-domain DOMAIN [PAGE >= 1]"++-- Empty relative names refer to the apex. Wildcards remain literal.+printRecord :: Text.Text -> DnsRecord -> IO ()+printRecord domain record = putStrLn (intercalate "\t"+  [ Text.unpack (if Text.null (dnsSubdomain record) then domain+      else dnsSubdomain record <> Text.pack "." <> domain)+  , Text.unpack (dnsType record)+  , Text.unpack (dnsValue record)+  , Text.unpack (dnsLastSeen record)+  ])++renderNames :: [Text.Text] -> String+renderNames []     = "[]"+renderNames values = intercalate "," (map Text.unpack values)
examples/search/Main.hs view
@@ -27,7 +27,7 @@               putStrLn ("Total: " ++ show (searchTotal found))               putStrLn ("Page: " ++ show page)               putStrLn ("Returned: " ++ show (length (searchMatches found)))-              putStrLn "IP\tPORT\tPRODUCT\tVERSION\tASN"+              putStrLn "IP\tPORT\tPRODUCT\tVERSION\tASN\tCOLLECTED_UTC\tHOSTNAMES\tDOMAINS\tALPN\tCERTIFICATE_NAMES"               mapM_ printMatch (searchMatches found)  parseArguments :: [String] -> IO (Text.Text, Int)@@ -46,4 +46,15 @@   , maybe "-" Text.unpack (matchProduct match)   , maybe "-" Text.unpack (matchVersion match)   , maybe "-" Text.unpack (matchAsn match)+  , maybe "-" Text.unpack (matchTimestamp match)+  , renderList (matchHostnames match)+  , renderList (matchDomains match)+  , renderList (matchAlpn match)+  , renderList (matchCertificateNames match)   ])++-- Keep absent data distinct from a reported empty list.+renderList :: Maybe [Text.Text] -> String+renderList Nothing       = "-"+renderList (Just [])     = "[]"+renderList (Just values) = intercalate "," (map Text.unpack values)
shodan.cabal view
@@ -1,12 +1,13 @@ cabal-version: 2.4 name: shodan-version: 0.3.0.0+version: 0.4.0.0 license: BSD-3-Clause license-file: LICENSE synopsis: A simple Haskell client for the Shodan API description:   A small Shodan API client using ordinary records and explicit IO functions.-  Supports API plan information, search result counts, and paginated searches,+  Supports API plan information, search result counts, paginated searches,+  and domain DNS information,   with JSON response types for facet statistics. Includes executable examples   and local HTTP tests. category: Network@@ -66,10 +67,20 @@   default-language: Haskell2010   ghc-options: -Wall -threaded +executable shodan-domain+  main-is: Main.hs+  hs-source-dirs: examples/domain+  build-depends:+    base >= 4.19 && < 5,+    shodan,+    text >= 2.1 && < 2.2+  default-language: Haskell2010+  ghc-options: -Wall -threaded+ test-suite shodan-test   type: exitcode-stdio-1.0   main-is: Main.hs-  other-modules: SearchTests+  other-modules: SearchTests, DnsTests   hs-source-dirs: test   build-depends:     base >= 4.19 && < 5,
src/Shodan.hs view
@@ -9,6 +9,9 @@   , getApiInfo   , countHosts   , searchHosts+  , getDomainInfo+  , DomainInfo (..)+  , DnsRecord (..)   , CountResult (..)   , Facet (..)   , SearchMatch (..)@@ -20,6 +23,8 @@ import           Control.Exception       (catch) import           Data.Aeson              (FromJSON (..), Value, eitherDecode',                                           withObject, (.!=), (.:!), (.:), (.:?))+import           Data.Aeson.Types        (Parser)+import           Data.Char               (chr, digitToInt, isHexDigit, ord) import           Data.Map.Strict         (Map) import qualified Data.Map.Strict         as Map import           Data.Text               (Text)@@ -54,7 +59,10 @@   | HttpError Int   | InvalidResponse   | InvalidPage-    -- ^ Search page numbers must be at least 1. No request was made.+    -- ^ Search and DNS page numbers must be at least 1. No request was made.+  | InvalidDomain+    -- ^ Expected an ASCII DNS name (use punycode for international names),+    -- without a URL, trailing dot, empty label, or leading/trailing hyphen.   deriving (Eq, Show)  -- | Retrieve @GET https://api.shodan.io/api-info@.@@ -82,6 +90,80 @@   | otherwise = getJSON client "/shodan/host/search"       [("query", query), ("page", DataText.pack (show page))] +-- | Retrieve one page of @GET /dns/domain/{domain}@. Shodan documents 100+-- records per page, but observed responses can exceed this; use 'domainMore'+-- rather than the record count to determine whether another page is available.+-- Uses one query credit per lookup according to the official API documentation.+-- Page numbers start at 1; additional pages are never fetched automatically.+-- Use 'domainMore' to decide whether to request the next page. The API defaults+-- are used: no historical records and no record-type filter.+-- Domain names must consist of ASCII labels of 1-63 letters, digits or hyphens,+-- with a total length of at most 253, and no leading/trailing label hyphens.+-- International names must be supplied in punycode; no DNS resolution is done.+-- Invalid pages or names are rejected before HTTP. Returned DNS data is an+-- observation by Shodan, not a guarantee of a current mapping or a complete list.+getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)+getDomainInfo client domain page+  | page < 1 = pure (Left InvalidPage)+  | not (validDomain domain) = pure (Left InvalidDomain)+  | otherwise = getJSON client ("/dns/domain/" <> domain)+      [("page", DataText.pack (show page))]++validDomain :: Text -> Bool+validDomain domain = DataText.length domain <= 253 &&+  all validLabel (DataText.splitOn "." domain)+  where+    validLabel label = not (DataText.null label) && DataText.length label <= 63 &&+      not (DataText.isPrefixOf "-" label || DataText.isSuffixOf "-" label) &&+      DataText.all (\c -> asciiLetter c || (c >= '0' && c <= '9') || c == '-') label+    asciiLetter c = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')++-- | One observed DNS record. All fields are required strings; unknown JSON+-- fields are ignored. Empty subdomains denote the apex domain. Wildcard names+-- are preserved. Type and value remain text for forward compatibility; neither+-- DNS syntax nor the timestamp format is validated during response parsing.+data DnsRecord = DnsRecord+  { dnsSubdomain :: Text+    -- ^ Relative name, for example @www@, @*.api@, or empty for the apex.+  , dnsType      :: Text+    -- ^ Record type such as @A@, @AAAA@, @CNAME@, @MX@, or @TXT@.+  , dnsValue     :: Text+    -- ^ Record value, preserved as returned rather than interpreted as an IP.+  , dnsLastSeen  :: Text+    -- ^ Shodan's last-observed timestamp, including its supplied timezone.+  } deriving (Eq, Show)++instance FromJSON DnsRecord where+  parseJSON = withObject "DnsRecord" $ \o -> DnsRecord+    <$> o .: "subdomain"+    <*> o .: "type"+    <*> o .: "value"+    <*> o .: "last_seen"++-- | A single page of domain information. All fields are required and unknown+-- fields are ignored. Lists preserve order, including wildcards and duplicates.+-- Empty lists are valid; missing/null lists and malformed records are rejected.+data DomainInfo = DomainInfo+  { domainName       :: Text+    -- ^ Domain as reported by Shodan.+  , domainTags       :: [Text]+    -- ^ Tags associated with this domain.+  , domainRecords    :: [DnsRecord]+    -- ^ DNS records on this page; not automatically combined with later pages.+  , domainSubdomains :: [Text]+    -- ^ Relative subdomain names returned by the API, not recomputed from records.+  , domainMore       :: Bool+    -- ^ Whether more DNS record pages are available.+  } deriving (Eq, Show)++instance FromJSON DomainInfo where+  parseJSON = withObject "DomainInfo" $ \o -> DomainInfo+    <$> o .: "domain"+    <*> o .: "tags"+    <*> o .: "data"+    <*> o .: "subdomains"+    <*> o .: "more"+ -- | Total matching results and optional breakdowns from @GET /shodan/host/count@. data CountResult = CountResult   { countTotal  :: Integer@@ -124,16 +206,31 @@ -- Unmodelled banner fields are ignored. The IP address is preserved as text; -- its syntax is not validated here. data SearchMatch = SearchMatch-  { matchIp      :: Text+  { matchIp               :: Text     -- ^ Required @ip_str@ field, usable for IPv4 or IPv6 text.-  , matchPort    :: Int+  , matchPort             :: Int     -- ^ Required integer port, from 0 through 65535.-  , matchProduct :: Maybe Text+  , matchProduct          :: Maybe Text     -- ^ Detected product; absent or null means 'Nothing'.-  , matchVersion :: Maybe Text+  , matchVersion          :: Maybe Text     -- ^ Detected product version; absent or null means 'Nothing'.-  , matchAsn     :: Maybe Text+  , matchAsn              :: Maybe Text     -- ^ ASN such as @AS65536@; absent or null means 'Nothing'.+  , matchTimestamp        :: Maybe Text+    -- ^ Banner collection time in UTC, preserved as supplied; not date-validated.+  , matchHostnames        :: Maybe [Text]+    -- ^ Hostnames based on reverse DNS; not an exhaustive list of hosted sites.+  , matchDomains          :: Maybe [Text]+    -- ^ Domains reported by Shodan, preserving order.+  , matchAlpn             :: Maybe [Text]+    -- ^ Observed @ssl.alpn@ protocols, such as @h2@ and @http/1.1@.+    -- Missing data does not imply that a protocol is unsupported.+  , matchCertificateNames :: Maybe [Text]+    -- ^ DNS names from the leaf certificate's @subjectAltName@ extension.+    -- Wildcards and order are preserved; IP, URI and email entries are omitted.+    -- These names are candidates, not verified DNS mappings or live sites.+    -- Missing certificate/extensions/SAN means 'Nothing'; a SAN without DNS+    -- entries yields @Just []@. Malformed SAN data rejects the response.   } deriving (Eq, Show)  instance FromJSON SearchMatch where@@ -143,9 +240,92 @@     productName <- o .:? "product"     version <- o .:? "version"     asn <- o .:? "asn"+    timestamp <- o .:? "timestamp"+    hostnames <- o .:? "hostnames"+    domains <- o .:? "domains"+    ssl <- o .:? "ssl"+    (alpn, certificateNames) <- case ssl of+      Nothing -> pure (Nothing, Nothing)+      Just value -> withObject "ssl" (\tls -> do+        protocols <- tls .:? "alpn"+        cert <- tls .:? "cert"+        names <- case cert of+          Nothing -> pure Nothing+          Just certificate -> withObject "certificate" (\c -> do+            extensions <- c .:? "extensions"+            case extensions of+              Nothing -> pure Nothing+              Just entries -> do+                decoded <- mapM parseCertificateExtension entries+                let sans = [names | Just names <- decoded]+                pure (if null sans then Nothing else Just (concat sans))) certificate+        pure (protocols, names)) value     if port < 0 || port > 65535       then fail "Search match port must be between 0 and 65535"-      else pure (SearchMatch ip port productName version asn)+      else pure (SearchMatch ip port productName version asn timestamp hostnames domains alpn certificateNames)++-- Shodan represents extension DER as text, sometimes with literal Python-style+-- escapes (as in its official example). Decode bytes before reading the SAN.+parseCertificateExtension :: Value -> Parser (Maybe [Text])+parseCertificateExtension value = withObject "certificate extension" (\extension -> do+  name <- extension .: "name"+  if name /= ("subjectAltName" :: Text)+    then pure Nothing+    else do+      raw <- extension .: "data"+      case decodeSan raw of+        Left message -> fail message+        Right names  -> pure (Just names)) value++decodeSan :: Text -> Either String [Text]+decodeSan raw = do+  bytes <- unescape (DataText.unpack raw)+  (tag, body, rest) <- tlv bytes+  if tag /= 48 || not (null rest)+    then Left "SAN must be a DER sequence"+    else names body+  where+    unescape [] = Right []+    unescape ('\\':'x':a:b:xs)+      | isHexDigit a && isHexDigit b =+          ((16 * digitToInt a + digitToInt b) :) <$> unescape xs+    unescape ('\\':c:xs) = case lookup c+        [('n',10), ('r',13), ('t',9), ('\\',92), ('\'',39), ('"',34)] of+      Just byte -> (byte :) <$> unescape xs+      Nothing   -> Left "Invalid escaped SAN byte"+    unescape (c:xs)+      | ord c <= 255 = (ord c :) <$> unescape xs+      | otherwise = Left "SAN contains a non-byte character"+    names [] = Right []+    names bytes = do+      (tag, body, rest) <- tlv bytes+      following <- names rest+      if tag == 130+        then if null body || any (> 127) body+          then Left "Invalid SAN DNS name"+          else pure (DataText.pack (map chr body) : following)+        else pure following++-- Only definite, minimally encoded DER lengths are accepted. Bound the length+-- octets before converting to Int, and check available input before splitting.+tlv :: [Int] -> Either String (Int, [Int], [Int])+tlv (tag:first:rest)+  | tag `mod` 32 == 31 = Left "Unsupported SAN tag"+  | first < 128 = takeBody first rest+  | otherwise =+      let count = first - 128+          (digits, remaining) = splitAt count rest+          size = foldl (\n byte -> n * 256 + toInteger byte) 0 digits+      in if count < 1 || count > 4 || length digits /= count ||+            take 1 digits == [0] || size < 128 || size > toInteger (length remaining)+        then Left "Invalid SAN DER length"+        else takeBody (fromInteger size) remaining+  where+    takeBody size bytes+      | size > length bytes = Left "Truncated SAN DER value"+      | otherwise = let (body, remaining) = splitAt size bytes+                    in Right (tag, body, remaining)+tlv _ = Left "Truncated SAN DER header"  -- | One page returned by @GET /shodan/host/search@. -- The total covers the whole query and may exceed this page's match count.
+ test/DnsTests.hs view
@@ -0,0 +1,54 @@+{-# LANGUAGE OverloadedStrings #-}++module DnsTests (dnsParsingTests) where++import           Control.Monad     (unless)+import           Data.Aeson        (Value (..), eitherDecode', encode, object,+                                    toJSON, (.=))+import qualified Data.Aeson.KeyMap as KeyMap+import           Shodan++-- Same response shape as the official /dns/domain example, with reserved names.+dnsParsingTests :: IO ()+dnsParsingTests = do+  let record = DnsRecord "www" "A" "192.0.2.1" "2026-11-21T01:02:03+00:00"+      wanted = DomainInfo "example.com" ["ipv6"] [record] ["www", "*.api"] True+      recordJSON = object ["subdomain" .= String "www", "type" .= String "A",+        "value" .= String "192.0.2.1", "last_seen" .= String "2026-11-21T01:02:03+00:00"]+      response = object ["domain" .= String "example.com", "tags" .= [String "ipv6"],+        "data" .= [recordJSON], "subdomains" .= [String "www", String "*.api"],+        "more" .= True, "future" .= Null]+      decode value = eitherDecode' (encode value) :: Either String DomainInfo+      check label ok = unless ok (fail label)+      reject value = check "invalid domain response accepted" (case decode value of+        Left _  -> True+        Right _ -> False)+      modifyRecord change = case response of+        Object o -> Object (KeyMap.insert "data" (toList (change recordJSON)) o)+        _        -> response+      toList value = toJSON [value]+      alter f (Object o) = Object (f o)+      alter _ value      = value+  check "domain response fields" (decode response == Right wanted)+  check "empty DNS lists" (eitherDecode'+    "{\"domain\":\"example.com\",\"tags\":[],\"data\":[],\"subdomains\":[],\"more\":false}" ==+    Right (DomainInfo "example.com" [] [] [] False))+  check "apex, unknown type, and timezone text preserved" (eitherDecode'+    "{\"subdomain\":\"\",\"type\":\"FUTURE\",\"value\":\"opaque value\",\"last_seen\":\"observed\",\"extra\":true}" ==+    Right (DnsRecord "" "FUTURE" "opaque value" "observed"))+  check "CNAME and wildcard preserved" (eitherDecode'+    "{\"subdomain\":\"*.api\",\"type\":\"CNAME\",\"value\":\"www.example.net\",\"last_seen\":\"2026-11-21T01:02:03Z\"}" ==+    Right (DnsRecord "*.api" "CNAME" "www.example.net" "2026-11-21T01:02:03Z"))+  mapM_ (\field -> do+    reject (alter (KeyMap.delete field) response)+    reject (alter (KeyMap.insert field Null) response))+    ["domain", "tags", "data", "subdomains", "more"]+  mapM_ (\field -> do+    reject (modifyRecord (alter (KeyMap.delete field)))+    reject (modifyRecord (alter (KeyMap.insert field Null)))+    reject (modifyRecord (alter (KeyMap.insert field (Number 42)))))+    ["subdomain", "type", "value", "last_seen"]+  mapM_ (\(field, value) -> reject (alter (KeyMap.insert field value) response))+    [("domain", Bool True), ("tags", String "ipv6"), ("data", object []),+     ("subdomains", toList (Number 42)), ("more", String "true")]+  reject (modifyRecord (const (String "not an object")))
test/Main.hs view
@@ -12,6 +12,7 @@ import qualified Data.Map.Strict          as Map import qualified Data.Text                as Text import qualified Data.Text.Encoding       as TextEncoding+import           DnsTests                 (dnsParsingTests) import qualified Network.HTTP.Client      as HTTP import qualified Network.HTTP.Types       as HTTPTypes import qualified Network.Wai              as Wai@@ -37,9 +38,11 @@   rejects "non-object response" (Array mempty)   facetTests   countParsingTests+  dnsParsingTests   searchParsingTests   httpTests   countTests+  domainHttpTests   searchHttpTests   putStrLn "All response parsing and HTTP tests passed." @@ -188,7 +191,7 @@       query = "http.title:\"한글 + & ?\" product:nginx"       responseBody = "{\"total\":101,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS65536\"}]}"       expected = Right (SearchResult 101-        [SearchMatch "192.0.2.1" 443 (Just "nginx") Nothing (Just "AS65536")] Map.empty)+        [SearchMatch "192.0.2.1" 443 (Just "nginx") Nothing (Just "AS65536") Nothing Nothing Nothing Nothing Nothing] Map.empty)       application page request respond = do         check "search GET" (Wai.requestMethod request == "GET")         check "search path" (Wai.pathInfo request == ["shodan", "host", "search"])@@ -223,6 +226,58 @@     withMockClient key mock $ \client -> do       actual <- searchHosts client query 1       check label (actual == wanted)) cases++domainHttpTests :: IO ()+domainHttpTests = do+  let key = "dns-key+&한글"+      body = "{\"domain\":\"Example.COM\",\"tags\":[],\"data\":[],\"subdomains\":[\"www\"],\"more\":true}"+      expected = Right (DomainInfo "Example.COM" [] [] ["www"] True)+      application page request respond = do+        check "domain GET" (Wai.requestMethod request == "GET")+        check "domain path" (Wai.rawPathInfo request == "/dns/domain/Example.COM")+        check "domain parameters" (Wai.queryString request ==+          [("key", Just (TextEncoding.encodeUtf8 key)), ("page", Just page)])+        check "domain accepts JSON" (lookup "Accept" (Wai.requestHeaders request) == Just "application/json")+        respond (Wai.responseLBS HTTPTypes.status200 [] body)+  mapM_ (\(page, encoded) -> withMockClient key (application encoded) $ \client -> do+    actual <- getDomainInfo client "Example.COM" page+    check "domain selected page" (actual == expected)) [(1,"1"), (2,"2")]+  manager <- HTTP.newManager HTTP.defaultManagerSettings+    { HTTP.managerModifyRequest = \_ -> fail "Invalid DNS argument sent HTTP" }+  let client = newClientWithManager manager key+  mapM_ (\page -> do+    actual <- getDomainInfo client "example.com" page+    check "invalid DNS page" (actual == Left InvalidPage)) [0,-1]+  mapM_ (\domain -> do+    actual <- getDomainInfo client domain 1+    check "invalid DNS domain" (actual == Left InvalidDomain))+    ["", " ", ".", "example..com", "example.com.", "-bad.com", "bad-.com",+     "https://example.com", "example.com/path", "example.com?key=oops",+     "example.com#fragment", "example%2f.com", "example_com", "한글.com",+     Text.replicate 64 "a" <> ".com", Text.intercalate "." (replicate 4 (Text.replicate 63 "a"))]+  -- Longest allowed labels and complete name are accepted without modification.+  let longest = Text.intercalate "." (replicate 3 (Text.replicate 63 "a") ++ [Text.replicate 61 "b"])+  mapM_ (\domain -> do+    let mock request respond = do+          check "DNS name passed verbatim" (Wai.pathInfo request == ["dns", "domain", domain])+          respond (Wai.responseLBS HTTPTypes.status200 [] body)+    withMockClient key mock $ \validClient -> do+      actual <- getDomainInfo validClient domain 1+      check "valid DNS name" (actual == expected)) ["xn--3e0b707e.com", "a-b.example.com", longest]+  let cases =+        [(HTTPTypes.status401, "secret", Left (HttpError 401)),+         (HTTPTypes.status403, "secret", Left (HttpError 403)),+         (HTTPTypes.status404, "secret", Left (HttpError 404)),+         (HTTPTypes.status429, "secret", Left (HttpError 429)),+         (HTTPTypes.status500, "secret", Left (HttpError 500)),+         (HTTPTypes.status302, "redirect", Left (HttpError 302)),+         (HTTPTypes.status200, "not JSON", Left InvalidResponse),+         (HTTPTypes.status200, "{}", Left InvalidResponse)]+  mapM_ (\(status, responseBody, wanted) -> do+    let mock _ respond = respond (Wai.responseLBS status [("Location", "/unexpected")] responseBody)+    withMockClient key mock $ \validClient -> do+      actual <- getDomainInfo validClient "example.com" 1+      check "domain HTTP error" (actual == wanted)) cases  countResponse :: String -> HTTPTypes.Status -> Lazy.ByteString   -> Either ShodanError CountResult -> IO ()
test/SearchTests.hs view
@@ -12,7 +12,7 @@ searchParsingTests = do   -- Selected fields from the official GET /shodan/host/search example.   -- Source: https://developer.shodan.io/api-  let match = SearchMatch "96.93.212.27" 443 (Just "nginx") Nothing (Just "AS7922")+  let match = SearchMatch "96.93.212.27" 443 (Just "nginx") Nothing (Just "AS7922") Nothing Nothing Nothing Nothing Nothing       expected = SearchResult 23047224 [match]         (Map.singleton "country" [Facet (String "US") 7883733])       example = "{\"total\":23047224,\"matches\":[{\"ip_str\":\"96.93.212.27\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS7922\",\"http\":{\"server\":\"nginx\"}}],\"facets\":{\"country\":[{\"value\":\"US\",\"count\":7883733}]}}"@@ -24,18 +24,60 @@     Right (SearchResult 100000000000000000000 [] Map.empty))   check "product version" (eitherDecode'     "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"version\":\"1.26.2\"}" ==-    Right (SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "1.26.2") Nothing))-  let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing Nothing+    Right (SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "1.26.2") Nothing Nothing Nothing Nothing Nothing Nothing))+  check "complete metadata" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":\"2026-11-21T01:02:03.123456\",\"hostnames\":[\"www.example.com\",\"api.example.com\"],\"domains\":[\"example.com\"],\"ssl\":{\"alpn\":[\"h2\",\"http/1.1\"],\"future\":true}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing (Just "2026-11-21T01:02:03.123456") (Just ["www.example.com", "api.example.com"]) (Just ["example.com"]) (Just ["h2", "http/1.1"]) Nothing))+  check "empty metadata lists" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":[],\"domains\":[],\"ssl\":{\"alpn\":[]}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing (Just []) (Just []) (Just []) Nothing))+  check "null metadata" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":null,\"hostnames\":null,\"domains\":null,\"ssl\":null}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+  check "missing ALPN in ssl" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+  check "null ALPN" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":null}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+  check "SAN DNS names and wildcard" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0&\\\\x82\\\\x0fwww.example.com\\\\x82\\\\x0d*.example.com\\\\x87\\\\x04abcd\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just ["www.example.com", "*.example.com"])))+  check "raw DER SAN" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0&\\u0082\\u000fwww.example.com\\u0082\\r*.example.com\\u0087\\u0004abcd\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just ["www.example.com", "*.example.com"])))+  check "empty SAN" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x00\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just [])))+  check "IP-only SAN" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x06\\\\x87\\\\x04abcd\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just [])))+  check "absent SAN" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"keyUsage\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+  check "absent extensions" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+  check "null certificate" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":null}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+  check "null extensions" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":null}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+  check "long DER length" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x81\\\\x96\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\"}]}}}" ==+    Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just (replicate 10 "a.example.com"))))+  let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing   check "IPv6 and absent optional fields" (eitherDecode'     "{\"ip_str\":\"2001:db8::1\",\"port\":0}" == Right minimal)   check "null optional fields and maximum port" (eitherDecode'     "{\"ip_str\":\"192.0.2.1\",\"port\":65535,\"product\":null,\"version\":null,\"asn\":null}" ==-    Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing Nothing))+    Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))   let twoMatches = "{\"total\":2,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":80},{\"ip_str\":\"192.0.2.1\",\"port\":443}],\"facets\":{}}"   check "multiple services on one IP retain order" (eitherDecode' twoMatches ==     Right (SearchResult 2-      [SearchMatch "192.0.2.1" 80 Nothing Nothing Nothing,-       SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing] Map.empty))+      [SearchMatch "192.0.2.1" 80 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing,+       SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing] Map.empty))   mapM_ rejectsMatch     [ ("missing IP", "{\"port\":443}")     , ("null IP", "{\"ip_str\":null,\"port\":443}")@@ -49,6 +91,29 @@     , ("numeric version", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"version\":126}")     , ("object version", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"version\":{}}")     , ("wrong ASN type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"asn\":65536}")+    , ("numeric timestamp", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":42}")+    , ("object timestamp", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":{}}")+    , ("string hostnames", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":\"example.com\"}")+    , ("non-text hostname", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":[null]}")+    , ("object domains", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"domains\":{}}")+    , ("non-text domain", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"domains\":[42]}")+    , ("non-object ssl", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":[]}")+    , ("string ssl", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":\"TLS\"}")+    , ("string ALPN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":\"h2\"}}")+    , ("non-text ALPN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":[\"h2\",3]}}")+    , ("non-object certificate", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":[]}}")+    , ("non-list extensions", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":{}}}}")+    , ("non-object extension", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[3]}}}")+    , ("missing extension name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{}]}}}")+    , ("missing SAN data", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\"}]}}}")+    , ("non-text SAN data", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":42}]}}}")+    , ("truncated SAN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x05\\\\x82\\\\x03ab\"}]}}}")+    , ("trailing SAN bytes", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x00x\"}]}}}")+    , ("invalid SAN escape", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"\\\\xZZ\"}]}}}")+    , ("indefinite length", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x80\"}]}}}")+    , ("nonminimal DER length", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x81\\\\x00\"}]}}}")+    , ("empty DNS name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x02\\\\x82\\\\x00\"}]}}}")+    , ("non-ASCII DNS name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x03\\\\x82\\\\x01\\\\xff\"}]}}}")     , ("non-object match", "[]")     ]   mapM_ rejectsSearch