shodan 0.3.0.0 → 0.4.0.0
raw patch · 10 files changed
+569/−22 lines, 10 filesnew-component:exe:shodan-domainPVP ok
version bump matches the API change (PVP)
API changes (from Hackage documentation)
+ Shodan: DnsRecord :: Text -> Text -> Text -> Text -> DnsRecord
+ Shodan: DomainInfo :: Text -> [Text] -> [DnsRecord] -> [Text] -> Bool -> DomainInfo
+ Shodan: InvalidDomain :: ShodanError
+ Shodan: [dnsLastSeen] :: DnsRecord -> Text
+ Shodan: [dnsSubdomain] :: DnsRecord -> Text
+ Shodan: [dnsType] :: DnsRecord -> Text
+ Shodan: [dnsValue] :: DnsRecord -> Text
+ Shodan: [domainMore] :: DomainInfo -> Bool
+ Shodan: [domainName] :: DomainInfo -> Text
+ Shodan: [domainRecords] :: DomainInfo -> [DnsRecord]
+ Shodan: [domainSubdomains] :: DomainInfo -> [Text]
+ Shodan: [domainTags] :: DomainInfo -> [Text]
+ Shodan: [matchAlpn] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchCertificateNames] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchDomains] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchHostnames] :: SearchMatch -> Maybe [Text]
+ Shodan: [matchTimestamp] :: SearchMatch -> Maybe Text
+ Shodan: data DnsRecord
+ Shodan: data DomainInfo
+ Shodan: getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.DnsRecord
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.DomainInfo
+ Shodan: instance GHC.Classes.Eq Shodan.DnsRecord
+ Shodan: instance GHC.Classes.Eq Shodan.DomainInfo
+ Shodan: instance GHC.Show.Show Shodan.DnsRecord
+ Shodan: instance GHC.Show.Show Shodan.DomainInfo
- Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> Maybe Text -> SearchMatch
+ Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> Maybe Text -> Maybe Text -> Maybe [Text] -> Maybe [Text] -> Maybe [Text] -> Maybe [Text] -> SearchMatch
Files
- CHANGELOG.md +31/−0
- README.md +87/−3
- examples/api-info/Main.hs +2/−1
- examples/domain/Main.hs +55/−0
- examples/search/Main.hs +12/−1
- shodan.cabal +14/−3
- src/Shodan.hs +187/−7
- test/DnsTests.hs +54/−0
- test/Main.hs +56/−1
- test/SearchTests.hs +71/−6
CHANGELOG.md view
@@ -2,6 +2,37 @@ This package follows the [Haskell Package Versioning Policy](https://pvp.haskell.org/). +## 0.4.0.0 — 2026-11-21++### Breaking changes++- Add `InvalidDomain` to `ShodanError`; exhaustive handlers need a new case.+- Append timestamp, hostname, domain, ALPN, and certificate DNS name fields to the public `SearchMatch`+ constructor. Positional construction and patterns need five additional fields;+ this datatype change requires a PVP major bump.++### Additions++- Add `getDomainInfo`, `DomainInfo`, and `DnsRecord` for a single page of domain+ DNS information, with explicit page selection and validation before HTTP.+- Add `shodan-domain DOMAIN [PAGE]`, including subdomains, records, last-seen+ timestamps, and the next-page flag. Each lookup uses one query credit;+ history and record-type filters are not implemented yet.+- Add domain parsing, HTTP, and argument validation tests and documentation.+ Verify a live domain lookup manually; use `domainMore` rather than assuming+ a fixed number of records per page.+- Add `matchTimestamp` (UTC collection time preserved as text), `matchHostnames`,+ `matchDomains`, and nested `ssl.alpn` as `matchAlpn`.+- Missing or null metadata becomes `Nothing`; empty lists remain `Just []`.+ Incorrect JSON types are rejected. Timestamp syntax is not validated.+- Show collection time, hostnames, domains, and ALPN in the search executable.+- Add `matchCertificateNames` and a `CERTIFICATE_NAMES` column for DNS names+ from the leaf certificate SAN, including wildcards. Decode DER and escaped+ byte representations; omit non-DNS entries and reject malformed SAN data.+ Certificate names are candidates, not verified DNS mappings or live sites.+- Add local parsing tests and document the meaning and limits of the metadata.+- Verify search metadata manually; automated tests remain local.+ ## 0.3.0.0 — 2026-11-20 ### Breaking changes
README.md view
@@ -4,12 +4,13 @@ Licensed under the [BSD 3-Clause License](LICENSE). -Current version: **0.3.0.0**. See the [changelog](CHANGELOG.md)+Current version: **0.4.0.0**. See the [changelog](CHANGELOG.md) for additions and the `SearchMatch` constructor change requiring a PVP major bump. This project supports `GET /api-info` (`ApiInfo` and `UsageLimits`), `GET /shodan/host/count` (`CountResult`), and one-page-`GET /shodan/host/search` (`SearchResult`). The API-info example fields and+`GET /shodan/host/search` (`SearchResult`), plus one-page+`GET /dns/domain/{domain}` (`DomainInfo`). The API-info example fields and the count response's `total` field are required; unknown fields are ignored. Numeric values use `Integer`; count totals must be nonnegative. @@ -152,10 +153,91 @@ required; the total is a nonnegative `Integer`. Facets follow the same parsing rules as `CountResult`. +Search metadata is also available through `matchTimestamp :: Maybe Text`,+`matchHostnames :: Maybe [Text]`, `matchDomains :: Maybe [Text]`, and+`matchAlpn :: Maybe [Text]`. The timestamp is the UTC banner collection time,+preserved as text without date validation. Hostnames are based on reverse DNS;+these lists do not enumerate all websites hosted on an IP.++`matchAlpn` reads the nested `ssl.alpn` list (for example `h2`, `http/1.1`).+It records Shodan's observation, not the protocol used for our API request.+Missing ALPN data does not establish that HTTP/2 or HTTP/3 is unsupported.+See the [banner specification](https://datapedia.shodan.io/) and+[TLS fields](https://datapedia.shodan.io/property/ssl.html).++All four metadata fields allow absent or null values as `Nothing`. Missing or+null `ssl` also produces `Nothing` for ALPN. A present non-null `ssl` must be an+object; non-text timestamps and lists containing non-text elements are rejected.+List order is preserved, and `Just []` is distinct from missing data.+The CLI adds `COLLECTED_UTC`, `HOSTNAMES`, `DOMAINS`, and `ALPN` columns.+Lists use commas, empty lists display `[]`, and missing values display `-`.++`matchCertificateNames :: Maybe [Text]` extracts DNS names from the leaf+certificate's `ssl.cert.extensions` entry named `subjectAltName` (SAN).+The search example displays them in `CERTIFICATE_NAMES`. Wildcards such as+`*.example.com` and list order are preserved; IP, URI, and email SAN entries+are omitted. These are certificate name candidates, not verified DNS mappings,+live sites, or a complete subdomain inventory. No DNS lookup or extra API call+is made, and the certificate common name is not used as a fallback.++Missing/null certificate or extensions, or no SAN extension, yields `Nothing`+(displayed as `-`). A present SAN containing no DNS names yields `Just []`+(displayed as `[]`). The parser handles DER bytes represented as JSON text and+literal Python-style byte escapes, as in the official Shodan example. Invalid+extension structure, non-text SAN data, and malformed DER reject the response.+Other extension contents are ignored. This reads names only; it does not validate+certificate signatures, expiry, or domain-name syntax.+ Optional facet requests and field selection are not supported yet. Search tests use local JSON fixtures and a mock HTTP server, and do not consume Shodan query credits. A one-page live search has also succeeded manually. +## Domain DNS information++Retrieve one page of Shodan's observed subdomains and DNS records:++```haskell+getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)+```++With `SHODAN_API_KEY` set, run:++```sh+cabal run shodan-domain -- example.com+cabal run shodan-domain -- example.com 2+```++The optional page defaults to 1. Shodan documents pages of 100 DNS records,+but live responses can contain more; use `domainMore`, not the record count,+to decide whether another page is available.+According to the [official DNS API documentation](https://developer.shodan.io/api),+each lookup uses one query credit. No extra pages or retries are automatic.+Check `domainMore` before requesting the next page. This endpoint queries+Shodan's stored DNS information, not a live DNS lookup, and cannot guarantee+that every subdomain is known or that a mapping is still current.++`DomainInfo` contains `domainName`, `domainTags`, `domainRecords`,+`domainSubdomains` (relative names as returned), and `domainMore`.+Each `DnsRecord` contains `dnsSubdomain`, `dnsType`, `dnsValue`, and+`dnsLastSeen`. All response fields are required; missing/null fields and wrong+JSON types are rejected, while unknown fields are ignored. List order and+wildcards are preserved. DNS types, values, and timestamps remain text without+syntax validation or IP-only assumptions. An empty record subdomain means the+apex domain. The CLI prints fully qualified record names, types, values, and+last-seen timestamps, plus relative subdomains and the `More` flag.++Supply an ASCII domain name (punycode for international names), without a URL+or trailing dot. Labels must have 1–63 letters, digits, or hyphens, without a+leading/trailing hyphen; the whole name must be at most 253 characters.+Invalid names return `Left InvalidDomain` and pages below 1 return+`Left InvalidPage`, before any HTTP request. The API key is handled through+the same client and redacted errors as the other endpoints.++This first implementation uses the API defaults: no history and no DNS+record-type filter. `/dns/resolve` and `/dns/reverse` are not implemented.+Automated DNS parsing and HTTP tests use fixtures and a local server; no+query credits are needed. A live domain lookup has also succeeded manually.+ ## Build The initial development compiler is GHC 9.8.4.@@ -186,7 +268,7 @@ ```sh stylish-haskell -i src/Shodan.hs test/*.hs examples/api-info/Main.hs stylish-haskell -i examples/count/Main.hs-stylish-haskell -i examples/search/Main.hs+stylish-haskell -i examples/search/Main.hs examples/domain/Main.hs cabal test ``` @@ -200,6 +282,8 @@ - `examples/count/Main.hs`: count-query executable using the same environment variable. - `examples/search/Main.hs`: one-page search executable with an optional page argument. - `test/Main.hs`: response parsing and HTTP tests against a local mock server.+- `examples/domain/Main.hs`: one-page domain DNS lookup executable.+- `test/DnsTests.hs`: domain and DNS record parsing tests. - `test/SearchTests.hs`: search-response JSON parsing tests. No API key is needed to build or test. Automated tests do not call Shodan.
examples/api-info/Main.hs view
@@ -28,4 +28,5 @@ errorMessage NetworkError = "Could not reach Shodan (connection failure or timeout)." errorMessage (HttpError status) = "Shodan returned HTTP " ++ show status ++ "." errorMessage InvalidResponse = "Shodan returned an unexpected response."-errorMessage InvalidPage = "Search page numbers must be at least 1."+errorMessage InvalidPage = "Page numbers must be at least 1."+errorMessage InvalidDomain = "Expected an ASCII DNS domain name without a URL or trailing dot."
+ examples/domain/Main.hs view
@@ -0,0 +1,55 @@+module Main (main) where++import Data.Char (isSpace)+import Data.List (intercalate)+import qualified Data.Text as Text+import Shodan (DnsRecord (..), DomainInfo (..),+ getDomainInfo, newClient)+import System.Environment (getArgs, lookupEnv)+import System.Exit (die)+import Text.Read (readMaybe)++main :: IO ()+main = do+ arguments <- getArgs+ (domain, page) <- parseArguments arguments+ configuredKey <- lookupEnv "SHODAN_API_KEY"+ case configuredKey of+ Nothing -> die "Set the SHODAN_API_KEY environment variable before running."+ Just key+ | all isSpace key -> die "SHODAN_API_KEY must not be empty or whitespace."+ | otherwise -> do+ client <- newClient (Text.pack key)+ result <- getDomainInfo client domain page+ case result of+ Left err -> die ("Shodan domain lookup failed: " ++ show err)+ Right found -> do+ putStrLn ("Domain: " ++ Text.unpack (domainName found))+ putStrLn ("Page: " ++ show page)+ putStrLn ("More: " ++ show (domainMore found))+ putStrLn ("Subdomains: " ++ renderNames (domainSubdomains found))+ putStrLn "HOSTNAME\tTYPE\tVALUE\tLAST_SEEN"+ mapM_ (printRecord (domainName found)) (domainRecords found)++parseArguments :: [String] -> IO (Text.Text, Int)+parseArguments [domain] = parseArguments [domain, "1"]+parseArguments [domain, pageText]+ | not (all isSpace domain)+ , Just page <- (readMaybe pageText :: Maybe Integer)+ , page >= 1+ , page <= toInteger (maxBound :: Int) = pure (Text.pack domain, fromInteger page)+parseArguments _ = die "Usage: shodan-domain DOMAIN [PAGE >= 1]"++-- Empty relative names refer to the apex. Wildcards remain literal.+printRecord :: Text.Text -> DnsRecord -> IO ()+printRecord domain record = putStrLn (intercalate "\t"+ [ Text.unpack (if Text.null (dnsSubdomain record) then domain+ else dnsSubdomain record <> Text.pack "." <> domain)+ , Text.unpack (dnsType record)+ , Text.unpack (dnsValue record)+ , Text.unpack (dnsLastSeen record)+ ])++renderNames :: [Text.Text] -> String+renderNames [] = "[]"+renderNames values = intercalate "," (map Text.unpack values)
examples/search/Main.hs view
@@ -27,7 +27,7 @@ putStrLn ("Total: " ++ show (searchTotal found)) putStrLn ("Page: " ++ show page) putStrLn ("Returned: " ++ show (length (searchMatches found)))- putStrLn "IP\tPORT\tPRODUCT\tVERSION\tASN"+ putStrLn "IP\tPORT\tPRODUCT\tVERSION\tASN\tCOLLECTED_UTC\tHOSTNAMES\tDOMAINS\tALPN\tCERTIFICATE_NAMES" mapM_ printMatch (searchMatches found) parseArguments :: [String] -> IO (Text.Text, Int)@@ -46,4 +46,15 @@ , maybe "-" Text.unpack (matchProduct match) , maybe "-" Text.unpack (matchVersion match) , maybe "-" Text.unpack (matchAsn match)+ , maybe "-" Text.unpack (matchTimestamp match)+ , renderList (matchHostnames match)+ , renderList (matchDomains match)+ , renderList (matchAlpn match)+ , renderList (matchCertificateNames match) ])++-- Keep absent data distinct from a reported empty list.+renderList :: Maybe [Text.Text] -> String+renderList Nothing = "-"+renderList (Just []) = "[]"+renderList (Just values) = intercalate "," (map Text.unpack values)
shodan.cabal view
@@ -1,12 +1,13 @@ cabal-version: 2.4 name: shodan-version: 0.3.0.0+version: 0.4.0.0 license: BSD-3-Clause license-file: LICENSE synopsis: A simple Haskell client for the Shodan API description: A small Shodan API client using ordinary records and explicit IO functions.- Supports API plan information, search result counts, and paginated searches,+ Supports API plan information, search result counts, paginated searches,+ and domain DNS information, with JSON response types for facet statistics. Includes executable examples and local HTTP tests. category: Network@@ -66,10 +67,20 @@ default-language: Haskell2010 ghc-options: -Wall -threaded +executable shodan-domain+ main-is: Main.hs+ hs-source-dirs: examples/domain+ build-depends:+ base >= 4.19 && < 5,+ shodan,+ text >= 2.1 && < 2.2+ default-language: Haskell2010+ ghc-options: -Wall -threaded+ test-suite shodan-test type: exitcode-stdio-1.0 main-is: Main.hs- other-modules: SearchTests+ other-modules: SearchTests, DnsTests hs-source-dirs: test build-depends: base >= 4.19 && < 5,
src/Shodan.hs view
@@ -9,6 +9,9 @@ , getApiInfo , countHosts , searchHosts+ , getDomainInfo+ , DomainInfo (..)+ , DnsRecord (..) , CountResult (..) , Facet (..) , SearchMatch (..)@@ -20,6 +23,8 @@ import Control.Exception (catch) import Data.Aeson (FromJSON (..), Value, eitherDecode', withObject, (.!=), (.:!), (.:), (.:?))+import Data.Aeson.Types (Parser)+import Data.Char (chr, digitToInt, isHexDigit, ord) import Data.Map.Strict (Map) import qualified Data.Map.Strict as Map import Data.Text (Text)@@ -54,7 +59,10 @@ | HttpError Int | InvalidResponse | InvalidPage- -- ^ Search page numbers must be at least 1. No request was made.+ -- ^ Search and DNS page numbers must be at least 1. No request was made.+ | InvalidDomain+ -- ^ Expected an ASCII DNS name (use punycode for international names),+ -- without a URL, trailing dot, empty label, or leading/trailing hyphen. deriving (Eq, Show) -- | Retrieve @GET https://api.shodan.io/api-info@.@@ -82,6 +90,80 @@ | otherwise = getJSON client "/shodan/host/search" [("query", query), ("page", DataText.pack (show page))] +-- | Retrieve one page of @GET /dns/domain/{domain}@. Shodan documents 100+-- records per page, but observed responses can exceed this; use 'domainMore'+-- rather than the record count to determine whether another page is available.+-- Uses one query credit per lookup according to the official API documentation.+-- Page numbers start at 1; additional pages are never fetched automatically.+-- Use 'domainMore' to decide whether to request the next page. The API defaults+-- are used: no historical records and no record-type filter.+-- Domain names must consist of ASCII labels of 1-63 letters, digits or hyphens,+-- with a total length of at most 253, and no leading/trailing label hyphens.+-- International names must be supplied in punycode; no DNS resolution is done.+-- Invalid pages or names are rejected before HTTP. Returned DNS data is an+-- observation by Shodan, not a guarantee of a current mapping or a complete list.+getDomainInfo :: Client -> Text -> Int -> IO (Either ShodanError DomainInfo)+getDomainInfo client domain page+ | page < 1 = pure (Left InvalidPage)+ | not (validDomain domain) = pure (Left InvalidDomain)+ | otherwise = getJSON client ("/dns/domain/" <> domain)+ [("page", DataText.pack (show page))]++validDomain :: Text -> Bool+validDomain domain = DataText.length domain <= 253 &&+ all validLabel (DataText.splitOn "." domain)+ where+ validLabel label = not (DataText.null label) && DataText.length label <= 63 &&+ not (DataText.isPrefixOf "-" label || DataText.isSuffixOf "-" label) &&+ DataText.all (\c -> asciiLetter c || (c >= '0' && c <= '9') || c == '-') label+ asciiLetter c = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')++-- | One observed DNS record. All fields are required strings; unknown JSON+-- fields are ignored. Empty subdomains denote the apex domain. Wildcard names+-- are preserved. Type and value remain text for forward compatibility; neither+-- DNS syntax nor the timestamp format is validated during response parsing.+data DnsRecord = DnsRecord+ { dnsSubdomain :: Text+ -- ^ Relative name, for example @www@, @*.api@, or empty for the apex.+ , dnsType :: Text+ -- ^ Record type such as @A@, @AAAA@, @CNAME@, @MX@, or @TXT@.+ , dnsValue :: Text+ -- ^ Record value, preserved as returned rather than interpreted as an IP.+ , dnsLastSeen :: Text+ -- ^ Shodan's last-observed timestamp, including its supplied timezone.+ } deriving (Eq, Show)++instance FromJSON DnsRecord where+ parseJSON = withObject "DnsRecord" $ \o -> DnsRecord+ <$> o .: "subdomain"+ <*> o .: "type"+ <*> o .: "value"+ <*> o .: "last_seen"++-- | A single page of domain information. All fields are required and unknown+-- fields are ignored. Lists preserve order, including wildcards and duplicates.+-- Empty lists are valid; missing/null lists and malformed records are rejected.+data DomainInfo = DomainInfo+ { domainName :: Text+ -- ^ Domain as reported by Shodan.+ , domainTags :: [Text]+ -- ^ Tags associated with this domain.+ , domainRecords :: [DnsRecord]+ -- ^ DNS records on this page; not automatically combined with later pages.+ , domainSubdomains :: [Text]+ -- ^ Relative subdomain names returned by the API, not recomputed from records.+ , domainMore :: Bool+ -- ^ Whether more DNS record pages are available.+ } deriving (Eq, Show)++instance FromJSON DomainInfo where+ parseJSON = withObject "DomainInfo" $ \o -> DomainInfo+ <$> o .: "domain"+ <*> o .: "tags"+ <*> o .: "data"+ <*> o .: "subdomains"+ <*> o .: "more"+ -- | Total matching results and optional breakdowns from @GET /shodan/host/count@. data CountResult = CountResult { countTotal :: Integer@@ -124,16 +206,31 @@ -- Unmodelled banner fields are ignored. The IP address is preserved as text; -- its syntax is not validated here. data SearchMatch = SearchMatch- { matchIp :: Text+ { matchIp :: Text -- ^ Required @ip_str@ field, usable for IPv4 or IPv6 text.- , matchPort :: Int+ , matchPort :: Int -- ^ Required integer port, from 0 through 65535.- , matchProduct :: Maybe Text+ , matchProduct :: Maybe Text -- ^ Detected product; absent or null means 'Nothing'.- , matchVersion :: Maybe Text+ , matchVersion :: Maybe Text -- ^ Detected product version; absent or null means 'Nothing'.- , matchAsn :: Maybe Text+ , matchAsn :: Maybe Text -- ^ ASN such as @AS65536@; absent or null means 'Nothing'.+ , matchTimestamp :: Maybe Text+ -- ^ Banner collection time in UTC, preserved as supplied; not date-validated.+ , matchHostnames :: Maybe [Text]+ -- ^ Hostnames based on reverse DNS; not an exhaustive list of hosted sites.+ , matchDomains :: Maybe [Text]+ -- ^ Domains reported by Shodan, preserving order.+ , matchAlpn :: Maybe [Text]+ -- ^ Observed @ssl.alpn@ protocols, such as @h2@ and @http/1.1@.+ -- Missing data does not imply that a protocol is unsupported.+ , matchCertificateNames :: Maybe [Text]+ -- ^ DNS names from the leaf certificate's @subjectAltName@ extension.+ -- Wildcards and order are preserved; IP, URI and email entries are omitted.+ -- These names are candidates, not verified DNS mappings or live sites.+ -- Missing certificate/extensions/SAN means 'Nothing'; a SAN without DNS+ -- entries yields @Just []@. Malformed SAN data rejects the response. } deriving (Eq, Show) instance FromJSON SearchMatch where@@ -143,9 +240,92 @@ productName <- o .:? "product" version <- o .:? "version" asn <- o .:? "asn"+ timestamp <- o .:? "timestamp"+ hostnames <- o .:? "hostnames"+ domains <- o .:? "domains"+ ssl <- o .:? "ssl"+ (alpn, certificateNames) <- case ssl of+ Nothing -> pure (Nothing, Nothing)+ Just value -> withObject "ssl" (\tls -> do+ protocols <- tls .:? "alpn"+ cert <- tls .:? "cert"+ names <- case cert of+ Nothing -> pure Nothing+ Just certificate -> withObject "certificate" (\c -> do+ extensions <- c .:? "extensions"+ case extensions of+ Nothing -> pure Nothing+ Just entries -> do+ decoded <- mapM parseCertificateExtension entries+ let sans = [names | Just names <- decoded]+ pure (if null sans then Nothing else Just (concat sans))) certificate+ pure (protocols, names)) value if port < 0 || port > 65535 then fail "Search match port must be between 0 and 65535"- else pure (SearchMatch ip port productName version asn)+ else pure (SearchMatch ip port productName version asn timestamp hostnames domains alpn certificateNames)++-- Shodan represents extension DER as text, sometimes with literal Python-style+-- escapes (as in its official example). Decode bytes before reading the SAN.+parseCertificateExtension :: Value -> Parser (Maybe [Text])+parseCertificateExtension value = withObject "certificate extension" (\extension -> do+ name <- extension .: "name"+ if name /= ("subjectAltName" :: Text)+ then pure Nothing+ else do+ raw <- extension .: "data"+ case decodeSan raw of+ Left message -> fail message+ Right names -> pure (Just names)) value++decodeSan :: Text -> Either String [Text]+decodeSan raw = do+ bytes <- unescape (DataText.unpack raw)+ (tag, body, rest) <- tlv bytes+ if tag /= 48 || not (null rest)+ then Left "SAN must be a DER sequence"+ else names body+ where+ unescape [] = Right []+ unescape ('\\':'x':a:b:xs)+ | isHexDigit a && isHexDigit b =+ ((16 * digitToInt a + digitToInt b) :) <$> unescape xs+ unescape ('\\':c:xs) = case lookup c+ [('n',10), ('r',13), ('t',9), ('\\',92), ('\'',39), ('"',34)] of+ Just byte -> (byte :) <$> unescape xs+ Nothing -> Left "Invalid escaped SAN byte"+ unescape (c:xs)+ | ord c <= 255 = (ord c :) <$> unescape xs+ | otherwise = Left "SAN contains a non-byte character"+ names [] = Right []+ names bytes = do+ (tag, body, rest) <- tlv bytes+ following <- names rest+ if tag == 130+ then if null body || any (> 127) body+ then Left "Invalid SAN DNS name"+ else pure (DataText.pack (map chr body) : following)+ else pure following++-- Only definite, minimally encoded DER lengths are accepted. Bound the length+-- octets before converting to Int, and check available input before splitting.+tlv :: [Int] -> Either String (Int, [Int], [Int])+tlv (tag:first:rest)+ | tag `mod` 32 == 31 = Left "Unsupported SAN tag"+ | first < 128 = takeBody first rest+ | otherwise =+ let count = first - 128+ (digits, remaining) = splitAt count rest+ size = foldl (\n byte -> n * 256 + toInteger byte) 0 digits+ in if count < 1 || count > 4 || length digits /= count ||+ take 1 digits == [0] || size < 128 || size > toInteger (length remaining)+ then Left "Invalid SAN DER length"+ else takeBody (fromInteger size) remaining+ where+ takeBody size bytes+ | size > length bytes = Left "Truncated SAN DER value"+ | otherwise = let (body, remaining) = splitAt size bytes+ in Right (tag, body, remaining)+tlv _ = Left "Truncated SAN DER header" -- | One page returned by @GET /shodan/host/search@. -- The total covers the whole query and may exceed this page's match count.
+ test/DnsTests.hs view
@@ -0,0 +1,54 @@+{-# LANGUAGE OverloadedStrings #-}++module DnsTests (dnsParsingTests) where++import Control.Monad (unless)+import Data.Aeson (Value (..), eitherDecode', encode, object,+ toJSON, (.=))+import qualified Data.Aeson.KeyMap as KeyMap+import Shodan++-- Same response shape as the official /dns/domain example, with reserved names.+dnsParsingTests :: IO ()+dnsParsingTests = do+ let record = DnsRecord "www" "A" "192.0.2.1" "2026-11-21T01:02:03+00:00"+ wanted = DomainInfo "example.com" ["ipv6"] [record] ["www", "*.api"] True+ recordJSON = object ["subdomain" .= String "www", "type" .= String "A",+ "value" .= String "192.0.2.1", "last_seen" .= String "2026-11-21T01:02:03+00:00"]+ response = object ["domain" .= String "example.com", "tags" .= [String "ipv6"],+ "data" .= [recordJSON], "subdomains" .= [String "www", String "*.api"],+ "more" .= True, "future" .= Null]+ decode value = eitherDecode' (encode value) :: Either String DomainInfo+ check label ok = unless ok (fail label)+ reject value = check "invalid domain response accepted" (case decode value of+ Left _ -> True+ Right _ -> False)+ modifyRecord change = case response of+ Object o -> Object (KeyMap.insert "data" (toList (change recordJSON)) o)+ _ -> response+ toList value = toJSON [value]+ alter f (Object o) = Object (f o)+ alter _ value = value+ check "domain response fields" (decode response == Right wanted)+ check "empty DNS lists" (eitherDecode'+ "{\"domain\":\"example.com\",\"tags\":[],\"data\":[],\"subdomains\":[],\"more\":false}" ==+ Right (DomainInfo "example.com" [] [] [] False))+ check "apex, unknown type, and timezone text preserved" (eitherDecode'+ "{\"subdomain\":\"\",\"type\":\"FUTURE\",\"value\":\"opaque value\",\"last_seen\":\"observed\",\"extra\":true}" ==+ Right (DnsRecord "" "FUTURE" "opaque value" "observed"))+ check "CNAME and wildcard preserved" (eitherDecode'+ "{\"subdomain\":\"*.api\",\"type\":\"CNAME\",\"value\":\"www.example.net\",\"last_seen\":\"2026-11-21T01:02:03Z\"}" ==+ Right (DnsRecord "*.api" "CNAME" "www.example.net" "2026-11-21T01:02:03Z"))+ mapM_ (\field -> do+ reject (alter (KeyMap.delete field) response)+ reject (alter (KeyMap.insert field Null) response))+ ["domain", "tags", "data", "subdomains", "more"]+ mapM_ (\field -> do+ reject (modifyRecord (alter (KeyMap.delete field)))+ reject (modifyRecord (alter (KeyMap.insert field Null)))+ reject (modifyRecord (alter (KeyMap.insert field (Number 42)))))+ ["subdomain", "type", "value", "last_seen"]+ mapM_ (\(field, value) -> reject (alter (KeyMap.insert field value) response))+ [("domain", Bool True), ("tags", String "ipv6"), ("data", object []),+ ("subdomains", toList (Number 42)), ("more", String "true")]+ reject (modifyRecord (const (String "not an object")))
test/Main.hs view
@@ -12,6 +12,7 @@ import qualified Data.Map.Strict as Map import qualified Data.Text as Text import qualified Data.Text.Encoding as TextEncoding+import DnsTests (dnsParsingTests) import qualified Network.HTTP.Client as HTTP import qualified Network.HTTP.Types as HTTPTypes import qualified Network.Wai as Wai@@ -37,9 +38,11 @@ rejects "non-object response" (Array mempty) facetTests countParsingTests+ dnsParsingTests searchParsingTests httpTests countTests+ domainHttpTests searchHttpTests putStrLn "All response parsing and HTTP tests passed." @@ -188,7 +191,7 @@ query = "http.title:\"한글 + & ?\" product:nginx" responseBody = "{\"total\":101,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS65536\"}]}" expected = Right (SearchResult 101- [SearchMatch "192.0.2.1" 443 (Just "nginx") Nothing (Just "AS65536")] Map.empty)+ [SearchMatch "192.0.2.1" 443 (Just "nginx") Nothing (Just "AS65536") Nothing Nothing Nothing Nothing Nothing] Map.empty) application page request respond = do check "search GET" (Wai.requestMethod request == "GET") check "search path" (Wai.pathInfo request == ["shodan", "host", "search"])@@ -223,6 +226,58 @@ withMockClient key mock $ \client -> do actual <- searchHosts client query 1 check label (actual == wanted)) cases++domainHttpTests :: IO ()+domainHttpTests = do+ let key = "dns-key+&한글"+ body = "{\"domain\":\"Example.COM\",\"tags\":[],\"data\":[],\"subdomains\":[\"www\"],\"more\":true}"+ expected = Right (DomainInfo "Example.COM" [] [] ["www"] True)+ application page request respond = do+ check "domain GET" (Wai.requestMethod request == "GET")+ check "domain path" (Wai.rawPathInfo request == "/dns/domain/Example.COM")+ check "domain parameters" (Wai.queryString request ==+ [("key", Just (TextEncoding.encodeUtf8 key)), ("page", Just page)])+ check "domain accepts JSON" (lookup "Accept" (Wai.requestHeaders request) == Just "application/json")+ respond (Wai.responseLBS HTTPTypes.status200 [] body)+ mapM_ (\(page, encoded) -> withMockClient key (application encoded) $ \client -> do+ actual <- getDomainInfo client "Example.COM" page+ check "domain selected page" (actual == expected)) [(1,"1"), (2,"2")]+ manager <- HTTP.newManager HTTP.defaultManagerSettings+ { HTTP.managerModifyRequest = \_ -> fail "Invalid DNS argument sent HTTP" }+ let client = newClientWithManager manager key+ mapM_ (\page -> do+ actual <- getDomainInfo client "example.com" page+ check "invalid DNS page" (actual == Left InvalidPage)) [0,-1]+ mapM_ (\domain -> do+ actual <- getDomainInfo client domain 1+ check "invalid DNS domain" (actual == Left InvalidDomain))+ ["", " ", ".", "example..com", "example.com.", "-bad.com", "bad-.com",+ "https://example.com", "example.com/path", "example.com?key=oops",+ "example.com#fragment", "example%2f.com", "example_com", "한글.com",+ Text.replicate 64 "a" <> ".com", Text.intercalate "." (replicate 4 (Text.replicate 63 "a"))]+ -- Longest allowed labels and complete name are accepted without modification.+ let longest = Text.intercalate "." (replicate 3 (Text.replicate 63 "a") ++ [Text.replicate 61 "b"])+ mapM_ (\domain -> do+ let mock request respond = do+ check "DNS name passed verbatim" (Wai.pathInfo request == ["dns", "domain", domain])+ respond (Wai.responseLBS HTTPTypes.status200 [] body)+ withMockClient key mock $ \validClient -> do+ actual <- getDomainInfo validClient domain 1+ check "valid DNS name" (actual == expected)) ["xn--3e0b707e.com", "a-b.example.com", longest]+ let cases =+ [(HTTPTypes.status401, "secret", Left (HttpError 401)),+ (HTTPTypes.status403, "secret", Left (HttpError 403)),+ (HTTPTypes.status404, "secret", Left (HttpError 404)),+ (HTTPTypes.status429, "secret", Left (HttpError 429)),+ (HTTPTypes.status500, "secret", Left (HttpError 500)),+ (HTTPTypes.status302, "redirect", Left (HttpError 302)),+ (HTTPTypes.status200, "not JSON", Left InvalidResponse),+ (HTTPTypes.status200, "{}", Left InvalidResponse)]+ mapM_ (\(status, responseBody, wanted) -> do+ let mock _ respond = respond (Wai.responseLBS status [("Location", "/unexpected")] responseBody)+ withMockClient key mock $ \validClient -> do+ actual <- getDomainInfo validClient "example.com" 1+ check "domain HTTP error" (actual == wanted)) cases countResponse :: String -> HTTPTypes.Status -> Lazy.ByteString -> Either ShodanError CountResult -> IO ()
test/SearchTests.hs view
@@ -12,7 +12,7 @@ searchParsingTests = do -- Selected fields from the official GET /shodan/host/search example. -- Source: https://developer.shodan.io/api- let match = SearchMatch "96.93.212.27" 443 (Just "nginx") Nothing (Just "AS7922")+ let match = SearchMatch "96.93.212.27" 443 (Just "nginx") Nothing (Just "AS7922") Nothing Nothing Nothing Nothing Nothing expected = SearchResult 23047224 [match] (Map.singleton "country" [Facet (String "US") 7883733]) example = "{\"total\":23047224,\"matches\":[{\"ip_str\":\"96.93.212.27\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS7922\",\"http\":{\"server\":\"nginx\"}}],\"facets\":{\"country\":[{\"value\":\"US\",\"count\":7883733}]}}"@@ -24,18 +24,60 @@ Right (SearchResult 100000000000000000000 [] Map.empty)) check "product version" (eitherDecode' "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"version\":\"1.26.2\"}" ==- Right (SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "1.26.2") Nothing))- let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing Nothing+ Right (SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "1.26.2") Nothing Nothing Nothing Nothing Nothing Nothing))+ check "complete metadata" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":\"2026-11-21T01:02:03.123456\",\"hostnames\":[\"www.example.com\",\"api.example.com\"],\"domains\":[\"example.com\"],\"ssl\":{\"alpn\":[\"h2\",\"http/1.1\"],\"future\":true}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing (Just "2026-11-21T01:02:03.123456") (Just ["www.example.com", "api.example.com"]) (Just ["example.com"]) (Just ["h2", "http/1.1"]) Nothing))+ check "empty metadata lists" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":[],\"domains\":[],\"ssl\":{\"alpn\":[]}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing (Just []) (Just []) (Just []) Nothing))+ check "null metadata" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":null,\"hostnames\":null,\"domains\":null,\"ssl\":null}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+ check "missing ALPN in ssl" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+ check "null ALPN" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":null}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing))+ check "SAN DNS names and wildcard" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0&\\\\x82\\\\x0fwww.example.com\\\\x82\\\\x0d*.example.com\\\\x87\\\\x04abcd\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just ["www.example.com", "*.example.com"])))+ check "raw DER SAN" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0&\\u0082\\u000fwww.example.com\\u0082\\r*.example.com\\u0087\\u0004abcd\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just ["www.example.com", "*.example.com"])))+ check "empty SAN" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x00\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just [])))+ check "IP-only SAN" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x06\\\\x87\\\\x04abcd\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just [])))+ check "absent SAN" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"keyUsage\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+ check "absent extensions" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+ check "null certificate" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":null}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+ check "null extensions" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":null}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Nothing)))+ check "long DER length" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x81\\\\x96\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\\\\x82\\\\x0da.example.com\"}]}}}" ==+ Right (SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing (Just (replicate 10 "a.example.com"))))+ let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing check "IPv6 and absent optional fields" (eitherDecode' "{\"ip_str\":\"2001:db8::1\",\"port\":0}" == Right minimal) check "null optional fields and maximum port" (eitherDecode' "{\"ip_str\":\"192.0.2.1\",\"port\":65535,\"product\":null,\"version\":null,\"asn\":null}" ==- Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing Nothing))+ Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing)) let twoMatches = "{\"total\":2,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":80},{\"ip_str\":\"192.0.2.1\",\"port\":443}],\"facets\":{}}" check "multiple services on one IP retain order" (eitherDecode' twoMatches == Right (SearchResult 2- [SearchMatch "192.0.2.1" 80 Nothing Nothing Nothing,- SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing] Map.empty))+ [SearchMatch "192.0.2.1" 80 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing,+ SearchMatch "192.0.2.1" 443 Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing] Map.empty)) mapM_ rejectsMatch [ ("missing IP", "{\"port\":443}") , ("null IP", "{\"ip_str\":null,\"port\":443}")@@ -49,6 +91,29 @@ , ("numeric version", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"version\":126}") , ("object version", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"version\":{}}") , ("wrong ASN type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"asn\":65536}")+ , ("numeric timestamp", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":42}")+ , ("object timestamp", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"timestamp\":{}}")+ , ("string hostnames", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":\"example.com\"}")+ , ("non-text hostname", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"hostnames\":[null]}")+ , ("object domains", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"domains\":{}}")+ , ("non-text domain", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"domains\":[42]}")+ , ("non-object ssl", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":[]}")+ , ("string ssl", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":\"TLS\"}")+ , ("string ALPN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":\"h2\"}}")+ , ("non-text ALPN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"alpn\":[\"h2\",3]}}")+ , ("non-object certificate", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":[]}}")+ , ("non-list extensions", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":{}}}}")+ , ("non-object extension", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[3]}}}")+ , ("missing extension name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{}]}}}")+ , ("missing SAN data", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\"}]}}}")+ , ("non-text SAN data", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":42}]}}}")+ , ("truncated SAN", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x05\\\\x82\\\\x03ab\"}]}}}")+ , ("trailing SAN bytes", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x00x\"}]}}}")+ , ("invalid SAN escape", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"\\\\xZZ\"}]}}}")+ , ("indefinite length", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x80\"}]}}}")+ , ("nonminimal DER length", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x81\\\\x00\"}]}}}")+ , ("empty DNS name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x02\\\\x82\\\\x00\"}]}}}")+ , ("non-ASCII DNS name", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"ssl\":{\"cert\":{\"extensions\":[{\"name\":\"subjectAltName\",\"data\":\"0\\\\x03\\\\x82\\\\x01\\\\xff\"}]}}}") , ("non-object match", "[]") ] mapM_ rejectsSearch