shodan 0.1.0.0 → 0.2.0.0
raw patch · 8 files changed
+339/−10 lines, 8 filesnew-component:exe:shodan-searchPVP ok
version bump matches the API change (PVP)
API changes (from Hackage documentation)
+ Shodan: InvalidPage :: ShodanError
+ Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> SearchMatch
+ Shodan: SearchResult :: Integer -> [SearchMatch] -> Map Text [Facet] -> SearchResult
+ Shodan: [matchAsn] :: SearchMatch -> Maybe Text
+ Shodan: [matchIp] :: SearchMatch -> Text
+ Shodan: [matchPort] :: SearchMatch -> Int
+ Shodan: [matchProduct] :: SearchMatch -> Maybe Text
+ Shodan: [searchFacets] :: SearchResult -> Map Text [Facet]
+ Shodan: [searchMatches] :: SearchResult -> [SearchMatch]
+ Shodan: [searchTotal] :: SearchResult -> Integer
+ Shodan: data SearchMatch
+ Shodan: data SearchResult
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.SearchMatch
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.SearchResult
+ Shodan: instance GHC.Classes.Eq Shodan.SearchMatch
+ Shodan: instance GHC.Classes.Eq Shodan.SearchResult
+ Shodan: instance GHC.Show.Show Shodan.SearchMatch
+ Shodan: instance GHC.Show.Show Shodan.SearchResult
+ Shodan: searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)
Files
- CHANGELOG.md +36/−0
- README.md +57/−5
- examples/api-info/Main.hs +1/−0
- examples/search/Main.hs +48/−0
- shodan.cabal +16/−4
- src/Shodan.hs +62/−1
- test/Main.hs +45/−0
- test/SearchTests.hs +74/−0
+ CHANGELOG.md view
@@ -0,0 +1,36 @@+# Changelog++This package follows the [Haskell Package Versioning Policy](https://pvp.haskell.org/).++## 0.2.0.0 — Unreleased++### Breaking changes++- Added `InvalidPage` to the public `ShodanError` type. Consumers that explicitly+ handle every constructor should add a case for this error. This datatype+ change requires a major version increase under PVP.++### Additions++- Added `searchHosts` for one page of `GET /shodan/host/search`, with an explicit+ page number starting at 1. Invalid page numbers are rejected before HTTP;+ additional pages are not fetched automatically.+- Added `SearchMatch` with IP, port, optional product, and optional ASN fields.+- Added `SearchResult` with total results, the current page of service matches,+ and optional facet breakdowns.+- Added the `shodan-search` executable, with an optional page argument and+ tabular service output. It reads `SHODAN_API_KEY` from the environment.+- Added search JSON parsing and local HTTP tests for page selection, query+ encoding, invalid pages, error responses, and malformed responses.+- Updated the README and Haddock documentation with search usage and credit+ behavior. Live search was also verified manually; automated tests remain local.++## 0.1.0.0 — 2026-11-20++- Initial Hackage release under the BSD 3-Clause License.+- Added API plan information and search result count requests.+- Added `ApiInfo`, `UsageLimits`, `CountResult`, and `Facet` response types.+- Added TLS client construction, redacted client display, HTTP error handling,+ and a finite response timeout without automatic redirect following.+- Added the `shodan-api-info` and `shodan-count` executable examples, local+ response parsing and HTTP tests, and Haddock documentation.
README.md view
@@ -4,8 +4,12 @@ Licensed under the [BSD 3-Clause License](LICENSE). -This project currently supports `GET /api-info` (`ApiInfo` and `UsageLimits`)-and `GET /shodan/host/count` (`CountResult`). The API-info example fields and+Version **0.2.0.0** is being prepared for release. See the [changelog](CHANGELOG.md)+for additions and the `ShodanError` constructor change requiring a PVP major bump.++This project supports `GET /api-info` (`ApiInfo` and `UsageLimits`),+`GET /shodan/host/count` (`CountResult`), and one-page+`GET /shodan/host/search` (`SearchResult`). The API-info example fields and the count response's `total` field are required; unknown fields are ignored. Numeric values use `Integer`; count totals must be nonnegative. @@ -75,8 +79,9 @@ countHosts :: Client -> Text -> IO (Either ShodanError CountResult) ``` -Read the total with `countTotal`. Optional facet requests and individual search-results are not supported yet. According to the [official count endpoint+Read the total with `countTotal`. For individual search results, use+`searchHosts` as described below. Optional facet requests are not supported yet.+According to the [official count endpoint documentation](https://developer.shodan.io/api), this call uses no query credits. ## Facet response type@@ -108,6 +113,49 @@ Both fields are required. Unknown fields are ignored, while missing fields, invalid counts, and non-object buckets are rejected. +## Search (0.2.0.0, unreleased)++Version 0.2.0.0 adds one-page searches:++```haskell+searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)+```++The page number starts at 1. Values below 1 return `Left InvalidPage` before+any request. Each call retrieves exactly the requested page; additional pages+are not fetched automatically. Shodan requires pages past 100 to be requested+in order.++With `SHODAN_API_KEY` set, the executable accepts a query and an optional page+(default 1):++```sh+cabal run shodan-search -- 'product:nginx country:KR'+cabal run shodan-search -- 'product:nginx country:KR' 2+```++These commands call Shodan and can use query credits: a filtered query or any+page after the first uses a credit under the [official API+policy](https://developer.shodan.io/api). The example prints the total across+all pages, the selected page number, the returned service count, and a table of+IP, port, product, and ASN. Missing product/ASN values are displayed as `-`.++`SearchMatch` describes one matching service with `matchIp`, `matchPort`,+`matchProduct`, and `matchAsn`. IP text and an integer port from 0 through 65535+are required. Product and ASN are optional: missing or null values become+`Nothing`, while other non-text values are rejected. IP syntax is not validated.+Unknown banner fields are ignored.++`SearchResult` contains `searchTotal`, `searchMatches`, and `searchFacets`.+The total covers all pages, while the matches describe one page and preserve+service order (multiple services may share an IP). Total and matches are+required; the total is a nonnegative `Integer`. Facets follow the same parsing+rules as `CountResult`.++Optional facet requests and field selection are not supported yet. Search+tests use local JSON fixtures and a mock HTTP server, and do not consume Shodan+query credits. A one-page live search has also succeeded manually.+ ## Build The initial development compiler is GHC 9.8.4.@@ -136,8 +184,9 @@ With the Cabal executable directory on `PATH`, run from the project directory: ```sh-stylish-haskell -i src/Shodan.hs test/Main.hs examples/api-info/Main.hs+stylish-haskell -i src/Shodan.hs test/*.hs examples/api-info/Main.hs stylish-haskell -i examples/count/Main.hs+stylish-haskell -i examples/search/Main.hs cabal test ``` @@ -145,10 +194,13 @@ - `shodan.cabal`: package and library settings. - `cabal.project`: local Cabal project settings.+- `CHANGELOG.md`: release history and compatibility notes. - `src/Shodan.hs`: client, errors, response records, and JSON parsing. - `examples/api-info/Main.hs`: executable example using an environment variable. - `examples/count/Main.hs`: count-query executable using the same environment variable.+- `examples/search/Main.hs`: one-page search executable with an optional page argument. - `test/Main.hs`: response parsing and HTTP tests against a local mock server.+- `test/SearchTests.hs`: search-response JSON parsing tests. No API key is needed to build or test. Automated tests do not call Shodan. They check request encoding, successful responses, HTTP errors, invalid JSON,
examples/api-info/Main.hs view
@@ -28,3 +28,4 @@ errorMessage NetworkError = "Could not reach Shodan (connection failure or timeout)." errorMessage (HttpError status) = "Shodan returned HTTP " ++ show status ++ "." errorMessage InvalidResponse = "Shodan returned an unexpected response."+errorMessage InvalidPage = "Search page numbers must be at least 1."
+ examples/search/Main.hs view
@@ -0,0 +1,48 @@+module Main (main) where++import Data.Char (isSpace)+import Data.List (intercalate)+import qualified Data.Text as Text+import Shodan (SearchMatch (..), SearchResult (..),+ newClient, searchHosts)+import System.Environment (getArgs, lookupEnv)+import System.Exit (die)+import Text.Read (readMaybe)++main :: IO ()+main = do+ arguments <- getArgs+ (query, page) <- parseArguments arguments+ configuredKey <- lookupEnv "SHODAN_API_KEY"+ case configuredKey of+ Nothing -> die "Set the SHODAN_API_KEY environment variable before running."+ Just key+ | all isSpace key -> die "SHODAN_API_KEY must not be empty or whitespace."+ | otherwise -> do+ client <- newClient (Text.pack key)+ result <- searchHosts client query page+ case result of+ Left err -> die ("Shodan search failed: " ++ show err)+ Right found -> do+ putStrLn ("Total: " ++ show (searchTotal found))+ putStrLn ("Page: " ++ show page)+ putStrLn ("Returned: " ++ show (length (searchMatches found)))+ putStrLn "IP\tPORT\tPRODUCT\tASN"+ mapM_ printMatch (searchMatches found)++parseArguments :: [String] -> IO (Text.Text, Int)+parseArguments [query] = parseArguments [query, "1"]+parseArguments [query, pageText]+ | not (all isSpace query)+ , Just page <- (readMaybe pageText :: Maybe Integer)+ , page >= 1+ , page <= toInteger (maxBound :: Int) = pure (Text.pack query, fromInteger page)+parseArguments _ = die "Usage: shodan-search 'QUERY' [PAGE >= 1]"++printMatch :: SearchMatch -> IO ()+printMatch match = putStrLn (intercalate "\t"+ [ Text.unpack (matchIp match)+ , show (matchPort match)+ , maybe "-" Text.unpack (matchProduct match)+ , maybe "-" Text.unpack (matchAsn match)+ ])
shodan.cabal view
@@ -1,13 +1,14 @@ cabal-version: 2.4 name: shodan-version: 0.1.0.0+version: 0.2.0.0 license: BSD-3-Clause license-file: LICENSE synopsis: A simple Haskell client for the Shodan API description: A small Shodan API client using ordinary records and explicit IO functions.- Supports API plan information and search result counts, with JSON response- types for facet statistics. Includes executable examples and local HTTP tests.+ Supports API plan information, search result counts, and paginated searches,+ with JSON response types for facet statistics. Includes executable examples+ and local HTTP tests. category: Network author: Joonkyu Park maintainer: vpark45@gmail.com@@ -15,7 +16,7 @@ bug-reports: https://github.com/nattybear/shodan/issues tested-with: GHC == 9.8.4 build-type: Simple-extra-source-files: README.md, .stylish-haskell.yaml+extra-source-files: README.md, CHANGELOG.md, .stylish-haskell.yaml source-repository head type: git@@ -55,9 +56,20 @@ default-language: Haskell2010 ghc-options: -Wall -threaded +executable shodan-search+ main-is: Main.hs+ hs-source-dirs: examples/search+ build-depends:+ base >= 4.19 && < 5,+ shodan,+ text >= 2.1 && < 2.2+ default-language: Haskell2010+ ghc-options: -Wall -threaded+ test-suite shodan-test type: exitcode-stdio-1.0 main-is: Main.hs+ other-modules: SearchTests hs-source-dirs: test build-depends: base >= 4.19 && < 5,
src/Shodan.hs view
@@ -8,18 +8,22 @@ , ShodanError (..) , getApiInfo , countHosts+ , searchHosts , CountResult (..) , Facet (..)+ , SearchMatch (..)+ , SearchResult (..) , ApiInfo (..) , UsageLimits (..) ) where import Control.Exception (catch) import Data.Aeson (FromJSON (..), Value, eitherDecode',- withObject, (.!=), (.:!), (.:))+ withObject, (.!=), (.:!), (.:), (.:?)) import Data.Map.Strict (Map) import qualified Data.Map.Strict as Map import Data.Text (Text)+import qualified Data.Text as DataText import qualified Data.Text.Encoding as Text import qualified Network.HTTP.Client as HTTP import Network.HTTP.Client.TLS (tlsManagerSettings)@@ -49,6 +53,8 @@ = NetworkError | HttpError Int | InvalidResponse+ | InvalidPage+ -- ^ Search page numbers must be at least 1. No request was made. deriving (Eq, Show) -- | Retrieve @GET https://api.shodan.io/api-info@.@@ -65,6 +71,17 @@ countHosts :: Client -> Text -> IO (Either ShodanError CountResult) countHosts client query = getJSON client "/shodan/host/count" [("query", query)] +-- | Retrieve one page from @GET /shodan/host/search@.+-- The page number starts at 1. No additional pages are fetched automatically.+-- Filtered queries and requests after page 1 use query credits. Requests past+-- page 100 must be made in order, as required by Shodan. Optional facets and+-- field selection are not requested.+searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)+searchHosts client query page+ | page < 1 = pure (Left InvalidPage)+ | otherwise = getJSON client "/shodan/host/search"+ [("query", query), ("page", DataText.pack (show page))]+ -- | Total matching results and optional breakdowns from @GET /shodan/host/count@. data CountResult = CountResult { countTotal :: Integer@@ -102,6 +119,50 @@ if count < 0 then fail "Facet count must not be negative" else pure (Facet value count)++-- | One matching service in a search response, rather than a unique host.+-- Unmodelled banner fields are ignored. The IP address is preserved as text;+-- its syntax is not validated here.+data SearchMatch = SearchMatch+ { matchIp :: Text+ -- ^ Required @ip_str@ field, usable for IPv4 or IPv6 text.+ , matchPort :: Int+ -- ^ Required integer port, from 0 through 65535.+ , matchProduct :: Maybe Text+ -- ^ Detected product; absent or null means 'Nothing'.+ , matchAsn :: Maybe Text+ -- ^ ASN such as @AS65536@; absent or null means 'Nothing'.+ } deriving (Eq, Show)++instance FromJSON SearchMatch where+ parseJSON = withObject "SearchMatch" $ \o -> do+ ip <- o .: "ip_str"+ port <- o .: "port"+ productName <- o .:? "product"+ asn <- o .:? "asn"+ if port < 0 || port > 65535+ then fail "Search match port must be between 0 and 65535"+ else pure (SearchMatch ip port productName asn)++-- | One page returned by @GET /shodan/host/search@.+-- The total covers the whole query and may exceed this page's match count.+data SearchResult = SearchResult+ { searchTotal :: Integer+ -- ^ Required nonnegative total across all pages.+ , searchMatches :: [SearchMatch]+ -- ^ Required service list; an empty page is valid. Order is preserved.+ , searchFacets :: Map Text [Facet]+ -- ^ Optional breakdowns. Missing means an empty map; null is rejected.+ } deriving (Eq, Show)++instance FromJSON SearchResult where+ parseJSON = withObject "SearchResult" $ \o -> do+ total <- o .: "total"+ matches <- o .: "matches"+ facets <- o .:! "facets" .!= Map.empty+ if total < 0+ then fail "Search total must not be negative"+ else pure (SearchResult total matches facets) -- Shared transport keeps status handling and key protection consistent. getJSON :: FromJSON a => Client -> Text -> [(Text, Text)]
test/Main.hs view
@@ -16,6 +16,7 @@ import qualified Network.HTTP.Types as HTTPTypes import qualified Network.Wai as Wai import Network.Wai.Handler.Warp (testWithApplication)+import SearchTests (searchParsingTests) import Shodan main :: IO ()@@ -36,8 +37,10 @@ rejects "non-object response" (Array mempty) facetTests countParsingTests+ searchParsingTests httpTests countTests+ searchHttpTests putStrLn "All response parsing and HTTP tests passed." facetTests :: IO ()@@ -178,6 +181,48 @@ (Left (HttpError 401)) countResponse "count rate limited" HTTPTypes.status429 "not JSON" (Left (HttpError 429))++searchHttpTests :: IO ()+searchHttpTests = do+ let key = "search-key+&"+ query = "http.title:\"한글 + & ?\" product:nginx"+ responseBody = "{\"total\":101,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS65536\"}]}"+ expected = Right (SearchResult 101+ [SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "AS65536")] Map.empty)+ application page request respond = do+ check "search GET" (Wai.requestMethod request == "GET")+ check "search path" (Wai.pathInfo request == ["shodan", "host", "search"])+ check "search encoded parameters" (Wai.queryString request ==+ [("key", Just (TextEncoding.encodeUtf8 key)),+ ("query", Just (TextEncoding.encodeUtf8 query)),+ ("page", Just page)])+ respond (Wai.responseLBS HTTPTypes.status200 [] responseBody)+ withMockClient key (application "1") $ \client -> do+ actual <- searchHosts client query 1+ check "first search page" (actual == expected)+ withMockClient key (application "2") $ \client -> do+ actual <- searchHosts client query 2+ check "explicit second search page" (actual == expected)+ manager <- HTTP.newManager HTTP.defaultManagerSettings+ { HTTP.managerModifyRequest = \_ -> fail "Invalid page sent an HTTP request" }+ mapM_ (\page -> do+ actual <- searchHosts (newClientWithManager manager key) query page+ check "invalid page rejected before HTTP" (actual == Left InvalidPage)) [0, -1]+ let cases =+ [ ("empty search page", HTTPTypes.status200, "{\"total\":0,\"matches\":[]}",+ Right (SearchResult 0 [] Map.empty))+ , ("search unauthorized", HTTPTypes.status401, "{\"error\":\"search-key+&\"}",+ Left (HttpError 401))+ , ("search rate limited", HTTPTypes.status429, "not JSON", Left (HttpError 429))+ , ("search invalid JSON", HTTPTypes.status200, "not JSON", Left InvalidResponse)+ , ("search invalid match", HTTPTypes.status200,+ "{\"total\":1,\"matches\":[{}]}", Left InvalidResponse)+ ]+ mapM_ (\(label, status, body, wanted) -> do+ let mock _ respond = respond (Wai.responseLBS status [] body)+ withMockClient key mock $ \client -> do+ actual <- searchHosts client query 1+ check label (actual == wanted)) cases countResponse :: String -> HTTPTypes.Status -> Lazy.ByteString -> Either ShodanError CountResult -> IO ()
+ test/SearchTests.hs view
@@ -0,0 +1,74 @@+{-# LANGUAGE OverloadedStrings #-}++module SearchTests (searchParsingTests) where++import Control.Monad (unless)+import Data.Aeson (Value (..), eitherDecode')+import qualified Data.ByteString.Lazy as Lazy+import qualified Data.Map.Strict as Map+import Shodan++searchParsingTests :: IO ()+searchParsingTests = do+ -- Selected fields from the official GET /shodan/host/search example.+ -- Source: https://developer.shodan.io/api+ let match = SearchMatch "96.93.212.27" 443 (Just "nginx") (Just "AS7922")+ expected = SearchResult 23047224 [match]+ (Map.singleton "country" [Facet (String "US") 7883733])+ example = "{\"total\":23047224,\"matches\":[{\"ip_str\":\"96.93.212.27\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS7922\",\"http\":{\"server\":\"nginx\"}}],\"facets\":{\"country\":[{\"value\":\"US\",\"count\":7883733}]}}"+ check "documented search fields" (eitherDecode' example == Right expected)+ check "empty search" (eitherDecode' "{\"total\":0,\"matches\":[]}" ==+ Right (SearchResult 0 [] Map.empty))+ check "total can exceed page size" (eitherDecode'+ "{\"total\":100000000000000000000,\"matches\":[],\"future\":true}" ==+ Right (SearchResult 100000000000000000000 [] Map.empty))+ let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing+ check "IPv6 and absent optional fields" (eitherDecode'+ "{\"ip_str\":\"2001:db8::1\",\"port\":0}" == Right minimal)+ check "null optional fields and maximum port" (eitherDecode'+ "{\"ip_str\":\"192.0.2.1\",\"port\":65535,\"product\":null,\"asn\":null}" ==+ Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing))+ let twoMatches = "{\"total\":2,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":80},{\"ip_str\":\"192.0.2.1\",\"port\":443}],\"facets\":{}}"+ check "multiple services on one IP retain order" (eitherDecode' twoMatches ==+ Right (SearchResult 2+ [SearchMatch "192.0.2.1" 80 Nothing Nothing,+ SearchMatch "192.0.2.1" 443 Nothing Nothing] Map.empty))+ mapM_ rejectsMatch+ [ ("missing IP", "{\"port\":443}")+ , ("null IP", "{\"ip_str\":null,\"port\":443}")+ , ("numeric IP", "{\"ip_str\":123,\"port\":443}")+ , ("missing port", "{\"ip_str\":\"192.0.2.1\"}")+ , ("null port", "{\"ip_str\":\"192.0.2.1\",\"port\":null}")+ , ("fractional port", "{\"ip_str\":\"192.0.2.1\",\"port\":1.5}")+ , ("negative port", "{\"ip_str\":\"192.0.2.1\",\"port\":-1}")+ , ("out-of-range port", "{\"ip_str\":\"192.0.2.1\",\"port\":65536}")+ , ("wrong product type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":true}")+ , ("wrong ASN type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"asn\":65536}")+ , ("non-object match", "[]")+ ]+ mapM_ rejectsSearch+ [ ("missing total", "{\"matches\":[]}")+ , ("negative total", "{\"total\":-1,\"matches\":[]}")+ , ("fractional total", "{\"total\":1.5,\"matches\":[]}")+ , ("missing matches", "{\"total\":0}")+ , ("null matches", "{\"total\":0,\"matches\":null}")+ , ("object matches", "{\"total\":0,\"matches\":{}}")+ , ("invalid nested match", "{\"total\":1,\"matches\":[{}]}")+ , ("null facets", "{\"total\":0,\"matches\":[],\"facets\":null}")+ , ("invalid nested facet", "{\"total\":0,\"matches\":[],\"facets\":{\"country\":[{}]}}")+ , ("non-object search", "[]")+ , ("malformed JSON", "not JSON")+ ]++check :: String -> Bool -> IO ()+check label success = unless success (fail label)++rejectsMatch :: (String, Lazy.ByteString) -> IO ()+rejectsMatch (label, body) = case eitherDecode' body :: Either String SearchMatch of+ Left _ -> pure ()+ Right _ -> fail (label ++ ": unexpectedly accepted")++rejectsSearch :: (String, Lazy.ByteString) -> IO ()+rejectsSearch (label, body) = case eitherDecode' body :: Either String SearchResult of+ Left _ -> pure ()+ Right _ -> fail (label ++ ": unexpectedly accepted")