superbuffer 0.1.0.0 → 0.2.0.0
raw patch · 8 files changed
+427/−218 lines, 8 filesdep +asyncdep +buffer-builderdep ~base
Dependencies added: async, buffer-builder
Dependency ranges changed: base
Files
- README.md +315/−123
- bench/Bench.hs +47/−11
- cbits/superbuffer.c +4/−0
- package.yaml +5/−2
- src/Data/ByteString/SuperBuffer.hs +21/−14
- stack.yaml +3/−64
- superbuffer.cabal +7/−4
- test/Test.hs +25/−0
README.md view
@@ -1,14 +1,14 @@ # Haskell SuperBuffer [](https://circleci.com/gh/agrafix/superbuffer)+[](http://hackage.haskell.org/package/superbuffer) The `superbuffer` packages was designed to efficiently build up bytestrings from `IO` actions producing smaller chunks. The goal was to reduce memory overhead as much as possible while still being as fast as possible. In our use case, it reduced total memory usage of the program from `350 MB` (`bytestring` builder) to `50 MB` (`superbuffer`). For speed see benchmarks below. Note that the speed heavily depends on a good choice of the initial buffer size and-the size of the chunks written. For small chunks the `superbuffer` outperforms the `bytestring` alternatives consistently. Note-that the library is currently not thread-safe, but it seems that it could be added (from outside) easily by making sure only one thread-at a time can call `appendBuffer` (making the call slower).+the size of the chunks written. For small chunks the `superbuffer` outperforms the `bytestring` alternatives consistently. `superbuffer`+also outperforms `buffer-builder`. ## Usage @@ -33,170 +33,362 @@ ``` benchmarking main/small/superbuffer (init=128 bytes)-time 22.20 ms (20.81 ms .. 24.69 ms)- 0.973 R² (0.946 R² .. 0.992 R²)-mean 22.47 ms (21.70 ms .. 23.45 ms)-std dev 1.905 ms (1.515 ms .. 2.685 ms)-variance introduced by outliers: 38% (moderately inflated)+time 20.69 ms (20.18 ms .. 21.25 ms)+ 0.997 R² (0.994 R² .. 0.999 R²)+mean 21.79 ms (21.35 ms .. 22.38 ms)+std dev 1.252 ms (699.1 μs .. 1.804 ms)+variance introduced by outliers: 23% (moderately inflated) benchmarking main/small/superbuffer (init=4000 bytes)-time 19.92 ms (19.28 ms .. 20.58 ms)- 0.996 R² (0.993 R² .. 0.998 R²)-mean 22.30 ms (21.56 ms .. 23.60 ms)-std dev 2.230 ms (1.324 ms .. 3.614 ms)-variance introduced by outliers: 46% (moderately inflated)+time 21.39 ms (20.63 ms .. 22.61 ms)+ 0.992 R² (0.984 R² .. 0.997 R²)+mean 22.40 ms (21.98 ms .. 22.86 ms)+std dev 997.0 μs (831.8 μs .. 1.181 ms)+variance introduced by outliers: 14% (moderately inflated) benchmarking main/small/superbuffer (init=8000 bytes)-time 22.72 ms (21.85 ms .. 23.92 ms)- 0.994 R² (0.989 R² .. 0.997 R²)-mean 23.25 ms (22.73 ms .. 23.95 ms)-std dev 1.496 ms (1.007 ms .. 2.375 ms)-variance introduced by outliers: 24% (moderately inflated)+time 20.94 ms (20.45 ms .. 21.41 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 21.80 ms (21.40 ms .. 22.64 ms)+std dev 1.249 ms (488.7 μs .. 1.935 ms)+variance introduced by outliers: 23% (moderately inflated) benchmarking main/small/superbuffer (init=16000 bytes)-time 23.28 ms (21.77 ms .. 25.14 ms)- 0.978 R² (0.953 R² .. 0.994 R²)-mean 23.41 ms (22.79 ms .. 24.34 ms)-std dev 1.727 ms (1.209 ms .. 2.455 ms)-variance introduced by outliers: 29% (moderately inflated)+time 21.86 ms (21.52 ms .. 22.29 ms)+ 0.998 R² (0.997 R² .. 0.999 R²)+mean 21.90 ms (21.64 ms .. 22.32 ms)+std dev 718.4 μs (447.8 μs .. 1.181 ms) benchmarking main/small/superbuffer (init=20000000 bytes)-time 35.78 ms (34.57 ms .. 37.21 ms)- 0.989 R² (0.972 R² .. 0.998 R²)-mean 36.45 ms (35.25 ms .. 37.34 ms)-std dev 2.167 ms (1.281 ms .. 3.677 ms)-variance introduced by outliers: 18% (moderately inflated)+time 38.11 ms (36.81 ms .. 39.41 ms)+ 0.995 R² (0.988 R² .. 0.998 R²)+mean 35.36 ms (33.46 ms .. 36.66 ms)+std dev 3.174 ms (2.018 ms .. 4.414 ms)+variance introduced by outliers: 36% (moderately inflated) +benchmarking main/small/superbuffer (init=128 bytes, threadsafe, 2 concurrent writes)+time 26.21 ms (25.50 ms .. 27.02 ms)+ 0.996 R² (0.992 R² .. 0.998 R²)+mean 25.38 ms (24.94 ms .. 25.83 ms)+std dev 1.015 ms (828.2 μs .. 1.274 ms)++benchmarking main/small/superbuffer (init=4000 bytes, threadsafe, 2 concurrent writes)+time 34.67 ms (33.62 ms .. 35.59 ms)+ 0.996 R² (0.990 R² .. 0.999 R²)+mean 32.54 ms (30.79 ms .. 33.70 ms)+std dev 2.867 ms (1.986 ms .. 3.938 ms)+variance introduced by outliers: 36% (moderately inflated)++benchmarking main/small/superbuffer (init=8000 bytes, threadsafe, 2 concurrent writes)+time 31.54 ms (30.29 ms .. 32.49 ms)+ 0.994 R² (0.988 R² .. 0.998 R²)+mean 31.80 ms (31.03 ms .. 32.62 ms)+std dev 1.600 ms (1.285 ms .. 2.193 ms)+variance introduced by outliers: 17% (moderately inflated)++benchmarking main/small/superbuffer (init=16000 bytes, threadsafe, 2 concurrent writes)+time 33.81 ms (32.98 ms .. 34.81 ms)+ 0.996 R² (0.991 R² .. 0.999 R²)+mean 30.72 ms (29.28 ms .. 31.86 ms)+std dev 2.803 ms (2.157 ms .. 3.648 ms)+variance introduced by outliers: 34% (moderately inflated)++benchmarking main/small/superbuffer (init=20000000 bytes, threadsafe, 2 concurrent writes)+time 18.96 ms (18.51 ms .. 19.47 ms)+ 0.997 R² (0.994 R² .. 0.999 R²)+mean 19.32 ms (19.05 ms .. 19.78 ms)+std dev 819.1 μs (465.1 μs .. 1.158 ms)+variance introduced by outliers: 13% (moderately inflated)++benchmarking main/small/buffer-builder (init=128 bytes, trim=yes)+time 28.90 ms (27.46 ms .. 30.66 ms)+ 0.991 R² (0.982 R² .. 0.999 R²)+mean 28.51 ms (27.96 ms .. 29.19 ms)+std dev 1.321 ms (914.7 μs .. 1.826 ms)+variance introduced by outliers: 16% (moderately inflated)++benchmarking main/small/buffer-builder (init=4000 bytes, trim=yes)+time 24.84 ms (24.26 ms .. 25.42 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 24.89 ms (24.64 ms .. 25.13 ms)+std dev 599.3 μs (498.5 μs .. 738.3 μs)++benchmarking main/small/buffer-builder (init=8000 bytes, trim=yes)+time 25.31 ms (24.89 ms .. 25.86 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 25.16 ms (24.89 ms .. 25.40 ms)+std dev 561.7 μs (441.5 μs .. 740.7 μs)++benchmarking main/small/buffer-builder (init=16000 bytes, trim=yes)+time 25.16 ms (24.64 ms .. 25.67 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 25.00 ms (24.71 ms .. 25.29 ms)+std dev 656.4 μs (502.2 μs .. 849.0 μs)++benchmarking main/small/buffer-builder (init=20000000 bytes, trim=yes)+time 26.82 ms (21.08 ms .. 36.36 ms)+ 0.808 R² (0.744 R² .. 0.999 R²)+mean 22.40 ms (21.28 ms .. 26.63 ms)+std dev 4.332 ms (512.5 μs .. 8.479 ms)+variance introduced by outliers: 72% (severely inflated)+ benchmarking main/small/bytestring builder-time 31.34 ms (28.98 ms .. 33.54 ms)- 0.987 R² (0.974 R² .. 0.999 R²)-mean 30.73 ms (29.54 ms .. 31.59 ms)-std dev 2.205 ms (1.699 ms .. 2.841 ms)-variance introduced by outliers: 28% (moderately inflated)+time 29.94 ms (27.48 ms .. 32.70 ms)+ 0.983 R² (0.968 R² .. 0.999 R²)+mean 31.64 ms (30.78 ms .. 32.14 ms)+std dev 1.435 ms (452.0 μs .. 2.120 ms)+variance introduced by outliers: 11% (moderately inflated) benchmarking main/small/bytestring fromChunks-time 26.35 ms (25.53 ms .. 27.40 ms)- 0.996 R² (0.991 R² .. 0.999 R²)-mean 26.35 ms (25.88 ms .. 27.79 ms)-std dev 1.685 ms (675.3 μs .. 3.068 ms)-variance introduced by outliers: 25% (moderately inflated)+time 26.12 ms (25.64 ms .. 26.60 ms)+ 0.998 R² (0.995 R² .. 0.999 R²)+mean 26.43 ms (26.02 ms .. 27.23 ms)+std dev 1.229 ms (671.2 μs .. 2.004 ms)+variance introduced by outliers: 15% (moderately inflated) benchmarking main/small/bytestring concat-time 30.54 ms (29.83 ms .. 31.13 ms)- 0.998 R² (0.997 R² .. 0.999 R²)-mean 31.45 ms (30.94 ms .. 33.55 ms)-std dev 1.658 ms (563.0 μs .. 3.317 ms)-variance introduced by outliers: 17% (moderately inflated)+time 26.74 ms (25.57 ms .. 28.26 ms)+ 0.993 R² (0.988 R² .. 0.999 R²)+mean 26.14 ms (25.72 ms .. 26.89 ms)+std dev 1.152 ms (450.9 μs .. 1.703 ms)+variance introduced by outliers: 15% (moderately inflated) benchmarking main/med/superbuffer (init=128 bytes)-time 20.98 ms (20.11 ms .. 21.97 ms)- 0.992 R² (0.983 R² .. 0.997 R²)-mean 23.11 ms (22.53 ms .. 24.12 ms)-std dev 1.686 ms (1.020 ms .. 2.914 ms)-variance introduced by outliers: 29% (moderately inflated)+time 23.30 ms (22.09 ms .. 24.39 ms)+ 0.991 R² (0.986 R² .. 0.996 R²)+mean 22.32 ms (21.82 ms .. 22.98 ms)+std dev 1.315 ms (966.3 μs .. 1.872 ms)+variance introduced by outliers: 24% (moderately inflated) benchmarking main/med/superbuffer (init=40000 bytes)-time 21.62 ms (20.92 ms .. 22.43 ms)- 0.996 R² (0.991 R² .. 0.998 R²)-mean 23.65 ms (23.06 ms .. 24.72 ms)-std dev 1.735 ms (1.044 ms .. 2.732 ms)-variance introduced by outliers: 29% (moderately inflated)+time 24.98 ms (24.71 ms .. 25.24 ms)+ 0.999 R² (0.999 R² .. 1.000 R²)+mean 25.30 ms (25.09 ms .. 25.70 ms)+std dev 609.2 μs (346.4 μs .. 1.017 ms) benchmarking main/med/superbuffer (init=80000 bytes)-time 20.65 ms (19.91 ms .. 21.35 ms)- 0.994 R² (0.985 R² .. 0.998 R²)-mean 22.64 ms (22.09 ms .. 23.84 ms)-std dev 1.767 ms (971.0 μs .. 3.008 ms)-variance introduced by outliers: 33% (moderately inflated)+time 20.93 ms (20.43 ms .. 21.26 ms)+ 0.998 R² (0.995 R² .. 0.999 R²)+mean 22.82 ms (22.30 ms .. 23.63 ms)+std dev 1.499 ms (1.115 ms .. 2.075 ms)+variance introduced by outliers: 24% (moderately inflated) benchmarking main/med/superbuffer (init=160000 bytes)-time 23.35 ms (22.15 ms .. 24.56 ms)- 0.992 R² (0.987 R² .. 0.998 R²)-mean 22.50 ms (22.15 ms .. 23.08 ms)-std dev 1.038 ms (639.2 μs .. 1.502 ms)-variance introduced by outliers: 14% (moderately inflated)+time 22.90 ms (22.48 ms .. 23.39 ms)+ 0.999 R² (0.997 R² .. 0.999 R²)+mean 23.07 ms (22.76 ms .. 23.32 ms)+std dev 647.4 μs (469.5 μs .. 928.6 μs) benchmarking main/med/superbuffer (init=20000000 bytes)-time 35.02 ms (31.20 ms .. 37.83 ms)- 0.983 R² (0.970 R² .. 0.998 R²)-mean 32.02 ms (31.12 ms .. 33.32 ms)-std dev 2.159 ms (1.456 ms .. 3.304 ms)-variance introduced by outliers: 24% (moderately inflated)+time 37.53 ms (36.30 ms .. 39.25 ms)+ 0.995 R² (0.990 R² .. 0.998 R²)+mean 33.78 ms (32.04 ms .. 35.11 ms)+std dev 3.209 ms (2.374 ms .. 4.178 ms)+variance introduced by outliers: 36% (moderately inflated) -benchmarking main/med/bytestring builder-time 22.03 ms (21.53 ms .. 22.48 ms)+benchmarking main/med/superbuffer (init=128 bytes, threadsafe, 2 concurrent writes)+time 23.38 ms (22.62 ms .. 24.39 ms)+ 0.995 R² (0.990 R² .. 0.999 R²)+mean 23.25 ms (22.88 ms .. 23.86 ms)+std dev 1.059 ms (684.7 μs .. 1.649 ms)+variance introduced by outliers: 14% (moderately inflated)++benchmarking main/med/superbuffer (init=40000 bytes, threadsafe, 2 concurrent writes)+time 20.71 ms (19.29 ms .. 21.60 ms)+ 0.984 R² (0.965 R² .. 0.997 R²)+mean 23.39 ms (22.47 ms .. 24.91 ms)+std dev 2.656 ms (1.445 ms .. 4.134 ms)+variance introduced by outliers: 53% (severely inflated)++benchmarking main/med/superbuffer (init=80000 bytes, threadsafe, 2 concurrent writes)+time 22.01 ms (21.56 ms .. 22.63 ms)+ 0.997 R² (0.993 R² .. 0.999 R²)+mean 22.14 ms (21.90 ms .. 22.51 ms)+std dev 701.7 μs (429.9 μs .. 927.8 μs)++benchmarking main/med/superbuffer (init=160000 bytes, threadsafe, 2 concurrent writes)+time 21.90 ms (21.60 ms .. 22.27 ms)+ 0.999 R² (0.998 R² .. 0.999 R²)+mean 22.06 ms (21.86 ms .. 22.31 ms)+std dev 508.7 μs (386.9 μs .. 661.0 μs)++benchmarking main/med/superbuffer (init=20000000 bytes, threadsafe, 2 concurrent writes)+time 16.40 ms (15.91 ms .. 16.82 ms)+ 0.997 R² (0.995 R² .. 0.999 R²)+mean 16.62 ms (16.41 ms .. 17.18 ms)+std dev 832.9 μs (303.1 μs .. 1.620 ms)+variance introduced by outliers: 20% (moderately inflated)++benchmarking main/med/buffer-builder (init=128 bytes, trim=yes)+time 28.20 ms (25.06 ms .. 30.59 ms)+ 0.975 R² (0.965 R² .. 0.988 R²)+mean 24.56 ms (23.74 ms .. 25.91 ms)+std dev 2.181 ms (1.301 ms .. 2.714 ms)+variance introduced by outliers: 35% (moderately inflated)++benchmarking main/med/buffer-builder (init=40000 bytes, trim=yes)+time 23.99 ms (23.69 ms .. 24.34 ms)+ 0.999 R² (0.998 R² .. 1.000 R²)+mean 23.72 ms (23.50 ms .. 23.97 ms)+std dev 531.5 μs (405.4 μs .. 771.5 μs)++benchmarking main/med/buffer-builder (init=80000 bytes, trim=yes)+time 21.96 ms (21.57 ms .. 22.41 ms) 0.998 R² (0.997 R² .. 0.999 R²)-mean 22.50 ms (22.06 ms .. 23.71 ms)-std dev 1.614 ms (467.3 μs .. 3.159 ms)-variance introduced by outliers: 29% (moderately inflated)+mean 22.18 ms (21.96 ms .. 22.49 ms)+std dev 639.7 μs (406.5 μs .. 1.025 ms) +benchmarking main/med/buffer-builder (init=160000 bytes, trim=yes)+time 22.29 ms (21.98 ms .. 22.60 ms)+ 0.999 R² (0.998 R² .. 1.000 R²)+mean 21.84 ms (21.61 ms .. 22.04 ms)+std dev 461.6 μs (367.1 μs .. 581.4 μs)++benchmarking main/med/buffer-builder (init=20000000 bytes, trim=yes)+time 39.52 ms (29.98 ms .. 47.38 ms)+ 0.876 R² (0.799 R² .. 0.949 R²)+mean 26.21 ms (23.50 ms .. 30.85 ms)+std dev 7.684 ms (4.775 ms .. 9.081 ms)+variance introduced by outliers: 88% (severely inflated)++benchmarking main/med/bytestring builder+time 21.52 ms (20.81 ms .. 22.22 ms)+ 0.996 R² (0.993 R² .. 0.999 R²)+mean 23.98 ms (23.12 ms .. 25.68 ms)+std dev 2.780 ms (1.529 ms .. 4.683 ms)+variance introduced by outliers: 53% (severely inflated)+ benchmarking main/med/bytestring fromChunks-time 23.88 ms (21.62 ms .. 26.14 ms)- 0.975 R² (0.954 R² .. 0.998 R²)-mean 22.81 ms (22.25 ms .. 23.95 ms)-std dev 1.812 ms (784.9 μs .. 2.692 ms)-variance introduced by outliers: 33% (moderately inflated)+time 24.67 ms (23.05 ms .. 26.69 ms)+ 0.981 R² (0.961 R² .. 0.994 R²)+mean 23.98 ms (23.31 ms .. 24.75 ms)+std dev 1.563 ms (1.232 ms .. 1.893 ms)+variance introduced by outliers: 25% (moderately inflated) benchmarking main/med/bytestring concat-time 21.79 ms (21.28 ms .. 22.23 ms)- 0.998 R² (0.997 R² .. 0.999 R²)-mean 22.52 ms (22.05 ms .. 23.70 ms)-std dev 1.646 ms (554.1 μs .. 3.042 ms)-variance introduced by outliers: 29% (moderately inflated)+time 22.40 ms (21.95 ms .. 22.86 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 23.13 ms (22.80 ms .. 24.16 ms)+std dev 1.202 ms (496.7 μs .. 2.131 ms)+variance introduced by outliers: 19% (moderately inflated) benchmarking main/large/superbuffer (init=128 bytes)-time 25.97 ms (25.22 ms .. 26.62 ms)- 0.994 R² (0.984 R² .. 0.999 R²)-mean 26.01 ms (25.18 ms .. 26.90 ms)-std dev 1.943 ms (1.274 ms .. 3.107 ms)-variance introduced by outliers: 30% (moderately inflated)+time 25.83 ms (25.33 ms .. 26.30 ms)+ 0.998 R² (0.997 R² .. 0.999 R²)+mean 26.00 ms (25.73 ms .. 26.33 ms)+std dev 662.6 μs (483.0 μs .. 977.9 μs) benchmarking main/large/superbuffer (init=400000 bytes)-time 21.51 ms (21.13 ms .. 21.88 ms)- 0.999 R² (0.997 R² .. 1.000 R²)-mean 22.14 ms (21.84 ms .. 22.75 ms)-std dev 939.7 μs (407.0 μs .. 1.669 ms)-variance introduced by outliers: 14% (moderately inflated)+time 21.47 ms (20.73 ms .. 22.38 ms)+ 0.995 R² (0.991 R² .. 0.998 R²)+mean 23.46 ms (22.95 ms .. 24.10 ms)+std dev 1.282 ms (979.1 μs .. 1.725 ms)+variance introduced by outliers: 19% (moderately inflated) benchmarking main/large/superbuffer (init=800000 bytes)-time 21.76 ms (21.35 ms .. 22.34 ms)- 0.997 R² (0.995 R² .. 0.999 R²)-mean 22.09 ms (21.77 ms .. 22.54 ms)-std dev 858.0 μs (532.7 μs .. 1.421 ms)-variance introduced by outliers: 14% (moderately inflated)+time 25.44 ms (25.11 ms .. 25.82 ms)+ 0.999 R² (0.998 R² .. 1.000 R²)+mean 25.43 ms (25.13 ms .. 25.93 ms)+std dev 816.9 μs (415.0 μs .. 1.352 ms) benchmarking main/large/superbuffer (init=1600000 bytes)-time 31.34 ms (30.55 ms .. 32.36 ms)- 0.997 R² (0.992 R² .. 0.999 R²)-mean 30.42 ms (29.34 ms .. 31.35 ms)-std dev 2.043 ms (1.383 ms .. 3.018 ms)-variance introduced by outliers: 22% (moderately inflated)+time 30.79 ms (30.24 ms .. 31.43 ms)+ 0.999 R² (0.997 R² .. 0.999 R²)+mean 30.44 ms (29.56 ms .. 30.92 ms)+std dev 1.449 ms (716.6 μs .. 2.528 ms)+variance introduced by outliers: 16% (moderately inflated) benchmarking main/large/superbuffer (init=20000000 bytes)-time 38.79 ms (36.27 ms .. 41.72 ms)- 0.983 R² (0.964 R² .. 0.995 R²)-mean 33.84 ms (32.32 ms .. 35.51 ms)-std dev 3.268 ms (2.816 ms .. 3.755 ms)-variance introduced by outliers: 36% (moderately inflated)+time 10.63 ms (4.936 ms .. 14.51 ms)+ 0.543 R² (0.201 R² .. 0.768 R²)+mean 27.28 ms (22.58 ms .. 32.47 ms)+std dev 10.98 ms (9.673 ms .. 12.05 ms)+variance introduced by outliers: 95% (severely inflated) -benchmarking main/large/bytestring builder-time 21.89 ms (21.35 ms .. 22.37 ms)+benchmarking main/large/superbuffer (init=128 bytes, threadsafe, 2 concurrent writes)+time 21.41 ms (20.05 ms .. 22.66 ms)+ 0.991 R² (0.986 R² .. 0.996 R²)+mean 24.39 ms (23.58 ms .. 25.36 ms)+std dev 2.007 ms (1.499 ms .. 2.498 ms)+variance introduced by outliers: 34% (moderately inflated)++benchmarking main/large/superbuffer (init=400000 bytes, threadsafe, 2 concurrent writes)+time 22.25 ms (21.85 ms .. 22.80 ms)+ 0.998 R² (0.996 R² .. 0.999 R²)+mean 22.25 ms (22.01 ms .. 22.49 ms)+std dev 553.6 μs (427.9 μs .. 702.3 μs)++benchmarking main/large/superbuffer (init=800000 bytes, threadsafe, 2 concurrent writes)+time 22.56 ms (22.14 ms .. 22.95 ms) 0.998 R² (0.995 R² .. 0.999 R²)-mean 23.02 ms (22.46 ms .. 24.92 ms)-std dev 2.169 ms (461.3 μs .. 4.197 ms)-variance introduced by outliers: 43% (moderately inflated)+mean 22.95 ms (22.64 ms .. 23.74 ms)+std dev 1.085 ms (445.3 μs .. 1.948 ms)+variance introduced by outliers: 14% (moderately inflated) +benchmarking main/large/superbuffer (init=1600000 bytes, threadsafe, 2 concurrent writes)+time 22.05 ms (21.75 ms .. 22.50 ms)+ 0.999 R² (0.997 R² .. 1.000 R²)+mean 22.77 ms (22.49 ms .. 23.59 ms)+std dev 1.020 ms (475.6 μs .. 1.879 ms)+variance introduced by outliers: 14% (moderately inflated)++benchmarking main/large/superbuffer (init=20000000 bytes, threadsafe, 2 concurrent writes)+time 31.35 ms (29.61 ms .. 33.21 ms)+ 0.986 R² (0.971 R² .. 0.995 R²)+mean 28.83 ms (27.87 ms .. 29.78 ms)+std dev 2.072 ms (1.698 ms .. 2.550 ms)+variance introduced by outliers: 28% (moderately inflated)++benchmarking main/large/buffer-builder (init=128 bytes, trim=yes)+time 27.56 ms (26.72 ms .. 28.08 ms)+ 0.996 R² (0.991 R² .. 0.999 R²)+mean 29.61 ms (28.91 ms .. 30.31 ms)+std dev 1.490 ms (1.282 ms .. 1.830 ms)+variance introduced by outliers: 16% (moderately inflated)++benchmarking main/large/buffer-builder (init=400000 bytes, trim=yes)+time 27.73 ms (26.91 ms .. 28.30 ms)+ 0.998 R² (0.995 R² .. 0.999 R²)+mean 27.06 ms (26.76 ms .. 27.36 ms)+std dev 675.1 μs (539.6 μs .. 851.8 μs)++benchmarking main/large/buffer-builder (init=800000 bytes, trim=yes)+time 27.74 ms (27.14 ms .. 28.16 ms)+ 0.999 R² (0.998 R² .. 1.000 R²)+mean 27.84 ms (27.57 ms .. 28.24 ms)+std dev 716.2 μs (406.4 μs .. 1.278 ms)++benchmarking main/large/buffer-builder (init=1600000 bytes, trim=yes)+time 27.90 ms (27.30 ms .. 28.26 ms)+ 0.999 R² (0.998 R² .. 1.000 R²)+mean 27.30 ms (27.01 ms .. 27.57 ms)+std dev 607.3 μs (472.2 μs .. 805.3 μs)++benchmarking main/large/buffer-builder (init=20000000 bytes, trim=yes)+time 40.14 ms (38.86 ms .. 41.10 ms)+ 0.998 R² (0.996 R² .. 1.000 R²)+mean 37.02 ms (33.52 ms .. 38.41 ms)+std dev 4.424 ms (1.138 ms .. 7.765 ms)+variance introduced by outliers: 45% (moderately inflated)++benchmarking main/large/bytestring builder+time 23.00 ms (21.84 ms .. 24.15 ms)+ 0.983 R² (0.960 R² .. 0.996 R²)+mean 24.99 ms (24.21 ms .. 26.16 ms)+std dev 2.223 ms (1.753 ms .. 3.120 ms)+variance introduced by outliers: 35% (moderately inflated)+ benchmarking main/large/bytestring fromChunks-time 21.91 ms (21.49 ms .. 22.31 ms)- 0.999 R² (0.998 R² .. 0.999 R²)-mean 22.47 ms (22.13 ms .. 23.63 ms)-std dev 1.230 ms (383.5 μs .. 2.326 ms)-variance introduced by outliers: 19% (moderately inflated)+time 23.39 ms (22.78 ms .. 23.99 ms)+ 0.998 R² (0.995 R² .. 0.999 R²)+mean 23.98 ms (23.58 ms .. 25.05 ms)+std dev 1.402 ms (420.3 μs .. 2.618 ms)+variance introduced by outliers: 24% (moderately inflated) benchmarking main/large/bytestring concat-time 22.12 ms (21.82 ms .. 22.42 ms)- 0.999 R² (0.998 R² .. 1.000 R²)-mean 22.60 ms (22.27 ms .. 23.51 ms)-std dev 1.198 ms (404.6 μs .. 2.244 ms)-variance introduced by outliers: 19% (moderately inflated)+time 29.05 ms (25.14 ms .. 32.45 ms)+ 0.960 R² (0.941 R² .. 0.990 R²)+mean 24.90 ms (23.74 ms .. 26.50 ms)+std dev 2.998 ms (1.942 ms .. 3.934 ms)+variance introduced by outliers: 51% (severely inflated) ```
bench/Bench.hs view
@@ -1,14 +1,15 @@ module Main where +import Control.Concurrent.Async import Control.Monad+import Criterion+import Criterion.Main import Data.ByteString.SuperBuffer import Data.Int+import qualified Data.BufferBuilder as BB import qualified Data.ByteString as BS-import qualified Data.ByteString.Lazy as BSL import qualified Data.ByteString.Builder as BSB--import Criterion-import Criterion.Main+import qualified Data.ByteString.Lazy as BSL main :: IO () main =@@ -22,18 +23,30 @@ mkGroup :: String -> Int -> Int -> Benchmark mkGroup name steps chunkSize =- bgroup name- [ bench (bufName iBufSize128) $ nfIO $ BS.reverse <$> buildBuf iBufSize128 steps chunkSize- , bench (bufName iBufSize) $ nfIO $ BS.reverse <$> buildBuf iBufSize steps chunkSize- , bench (bufName iBufSize2) $ nfIO $ BS.reverse <$> buildBuf iBufSize2 steps chunkSize- , bench (bufName iBufSize4) $ nfIO $ BS.reverse <$> buildBuf iBufSize4 steps chunkSize- , bench (bufName iBufSizeAll) $ nfIO $ BS.reverse <$> buildBuf iBufSizeAll steps chunkSize- , bench "bytestring builder" $ nfIO $ BS.reverse <$> buildBufBuilder steps chunkSize+ bgroup name $+ mkSizedGroup steps chunkSize bufName buildBuf+ ++ mkSizedGroup steps chunkSize bufNameT buildBufT+ ++ mkSizedGroup steps chunkSize bufBBName buildBufBB+ +++ [ bench "bytestring builder" $ nfIO $ BS.reverse <$> buildBufBuilder steps chunkSize , bench "bytestring fromChunks" $ nfIO $ BS.reverse <$> buildBufChunks steps chunkSize , bench "bytestring concat" $ nfIO $ BS.reverse <$> buildBufConcat steps chunkSize ] where+ bufBBName is = "buffer-builder (init=" ++ show is ++ " bytes, trim=yes)" bufName is = "superbuffer (init=" ++ show is ++ " bytes)"+ bufNameT is = "superbuffer (init=" ++ show is ++ " bytes, threadsafe, 2 concurrent writes)"++mkSizedGroup ::+ Int -> Int -> (Int64 -> String) -> (Int64 -> Int -> Int -> IO BS.ByteString) -> [Benchmark]+mkSizedGroup steps chunkSize bufName builder =+ [ bench (bufName iBufSize128) $ nfIO $ BS.reverse <$> builder iBufSize128 steps chunkSize+ , bench (bufName iBufSize) $ nfIO $ BS.reverse <$> builder iBufSize steps chunkSize+ , bench (bufName iBufSize2) $ nfIO $ BS.reverse <$> builder iBufSize2 steps chunkSize+ , bench (bufName iBufSize4) $ nfIO $ BS.reverse <$> builder iBufSize4 steps chunkSize+ , bench (bufName iBufSizeAll) $ nfIO $ BS.reverse <$> builder iBufSizeAll steps chunkSize+ ]+ where iBufSize128 = 128 iBufSize = fromIntegral chunkSize iBufSize2 = 2 * fromIntegral chunkSize@@ -50,6 +63,29 @@ withBuffer bufSize $ \buf -> forM_ [0..steps] $ \step -> appendBuffer buf (mkChunk step chunkSize)++buildBufT :: Int64 -> Int -> Int -> IO BS.ByteString+buildBufT bufSize steps chunkSize =+ withBuffer bufSize $ \buf ->+ forM_ [0..(ceiling halfSteps)] $ \step ->+ concurrently_+ (appendBufferT buf (mkChunk step chunkSize))+ (appendBufferT buf (mkChunk step chunkSize))+ where+ halfSteps :: Double+ halfSteps = fromIntegral steps / 2.0++buildBufBB :: Int64 -> Int -> Int -> IO BS.ByteString+buildBufBB bufSize steps chunkSize =+ pure $ BB.runBufferBuilderWithOptions opts $+ forM_ [0..steps] $ \step ->+ BB.appendBS (mkChunk step chunkSize)+ where+ opts =+ BB.Options+ { BB.initialCapacity = fromIntegral bufSize+ , BB.trimFinalBuffer = True+ } buildBufBuilder :: Int -> Int -> IO BS.ByteString buildBufBuilder steps chunkSize =
cbits/superbuffer.c view
@@ -39,6 +39,10 @@ char *read_sbuf(struct sbuf *buf, size_t *len) {+ if (buf->currentSize < buf->maxSize) {+ buf->contents = (char *)realloc(buf->contents, buf->currentSize);+ buf->maxSize = buf->currentSize; // not strictly needed, but for consistency+ } *len = buf->currentSize; return buf->contents; }
package.yaml view
@@ -1,5 +1,5 @@ name: superbuffer-version: 0.1.0.0+version: 0.2.0.0 synopsis: Efficiently build a bytestring from smaller chunks description: Efficiently (both fast and memory efficient) build a bytestring from smaller chunks homepage: https://github.com/agrafix/superbuffer#readme@@ -14,7 +14,7 @@ - package.yaml dependencies:- - base >= 4.7 && < 5+ - base >= 4.8 && < 5 - bytestring < 0.11 ghc-options: -Wall@@ -33,6 +33,7 @@ dependencies: - HTF < 0.14 - QuickCheck < 2.10+ - async - superbuffer ghc-options: -funfolding-use-threshold=16 -O2 -optc-Ofast @@ -43,4 +44,6 @@ dependencies: - criterion < 1.2 - superbuffer+ - buffer-builder+ - async ghc-options: -funfolding-use-threshold=16 -O2 -optc-Ofast
src/Data/ByteString/SuperBuffer.hs view
@@ -2,23 +2,23 @@ {-# LANGUAGE ForeignFunctionInterface #-} {-# LANGUAGE ScopedTypeVariables #-} module Data.ByteString.SuperBuffer- ( SuperBuffer, withBuffer, appendBuffer+ ( SuperBuffer, withBuffer, appendBuffer, appendBufferT ) where -import Foreign-import Foreign.C+import Control.Concurrent.MVar import Control.Exception- import Data.Coerce+import Foreign+import Foreign.C import qualified Data.ByteString as BS import qualified Data.ByteString.Unsafe as BS -- | The buffer. Internally only a pointer to a C struct. Don't worry, -- this module attempts to make usage of the SuperBuffer as safe as possible in--- terms of memory leaks (with exceptions).+-- terms of memory leaks (even when exceptions occur). newtype SuperBuffer- = SuperBuffer SuperBufferP+ = SuperBuffer (SuperBufferP, MVar ()) -- | Allocate a new buffer with a given initial size. The perfect starting point -- depends on the expected total size and the average size for a single chunk@@ -39,32 +39,39 @@ {-# INLINE withBuffer #-} newBuffer :: Int64 -> IO SuperBuffer-newBuffer size = SuperBuffer <$> new_sbuf (fromIntegral size)+newBuffer size = SuperBuffer <$> ((,) <$> new_sbuf (fromIntegral size) <*> newEmptyMVar) {-# INLINE newBuffer #-} -- | Write a bytestring to the buffer and grow the buffer if needed. Note that only--- one thread at any given time may call this function, so if you are sharing the--- 'SuperBuffer' between threads make sure you place some type of guarding/locking around--- this function.+-- one thread at any given time may call this function. Use 'appendBufferT' when+-- accessing 'SuperBuffer' from multiple threads. appendBuffer :: SuperBuffer -> BS.ByteString -> IO ()-appendBuffer (SuperBuffer ptr) bs =+appendBuffer (SuperBuffer (ptr, _)) bs = BS.unsafeUseAsCStringLen bs $ \(cstr, len) -> append_sbuf ptr cstr (fromIntegral len) {-# INLINE appendBuffer #-} +-- | Write a bytestring to the buffer and grow the buffer if needed. This function+-- can be used accross different threads, but is slower than 'appendBuffer'.+appendBufferT :: SuperBuffer -> BS.ByteString -> IO ()+appendBufferT buf@(SuperBuffer (_, lock)) bs =+ bracket_ (putMVar lock ()) (takeMVar lock) $+ appendBuffer buf bs+{-# INLINE appendBufferT #-}+ destroyBuffer :: SuperBuffer -> IO ()-destroyBuffer (SuperBuffer ptr) = destroy_sbuf ptr+destroyBuffer (SuperBuffer (ptr, _)) = destroy_sbuf ptr {-# INLINE destroyBuffer #-} destroyBufferContents :: SuperBuffer -> IO ()-destroyBufferContents (SuperBuffer ptr) = destroyContents_sbuf ptr+destroyBufferContents (SuperBuffer (ptr, _)) = destroyContents_sbuf ptr {-# INLINE destroyBufferContents #-} -- | Read the final buffer contents. This must only -- be called once readBuffer :: SuperBuffer -> IO BS.ByteString-readBuffer (SuperBuffer ptr) =+readBuffer (SuperBuffer (ptr, _)) = do (cstr, size) <- readLocal BS.unsafePackCStringFinalizer (coerce cstr) (fromIntegral size) (free cstr) where
stack.yaml view
@@ -1,66 +1,5 @@-# This file was automatically generated by 'stack init'-# -# Some commonly used options have been documented as comments in this file.-# For advanced use and comprehensive documentation of the format, please see:-# http://docs.haskellstack.org/en/stable/yaml_configuration/--# Resolver to choose a 'specific' stackage snapshot or a compiler version.-# A snapshot resolver dictates the compiler version and the set of packages-# to be used for project dependencies. For example:-# -# resolver: lts-3.5-# resolver: nightly-2015-09-21-# resolver: ghc-7.10.2-# resolver: ghcjs-0.1.0_ghc-7.10.2-# resolver:-# name: custom-snapshot-# location: "./custom-snapshot.yaml" resolver: lts-7.11--# User packages to be built.-# Various formats can be used as shown in the example below.-# -# packages:-# - some-directory-# - https://example.com/foo/bar/baz-0.0.2.tar.gz-# - location:-# git: https://github.com/commercialhaskell/stack.git-# commit: e7b331f14bcffb8367cd58fbfc8b40ec7642100a-# - location: https://github.com/commercialhaskell/stack/commit/e7b331f14bcffb8367cd58fbfc8b40ec7642100a-# extra-dep: true-# subdirs:-# - auto-update-# - wai-# -# A package marked 'extra-dep: true' will only be built if demanded by a-# non-dependency (i.e. a user package), and its test suites and benchmarks-# will not be run. This is useful for tweaking upstream packages. packages:-- '.'-# Dependency packages to be pulled from upstream that are not in the resolver-# (e.g., acme-missiles-0.3)-extra-deps: []--# Override default flag values for local packages and extra-deps-flags: {}--# Extra package databases containing global packages-extra-package-dbs: []--# Control whether we use the GHC we find on the path-# system-ghc: true-# -# Require a specific version of stack, using version ranges-# require-stack-version: -any # Default-# require-stack-version: ">=1.1"-# -# Override the architecture used by stack, especially useful on Windows-# arch: i386-# arch: x86_64-# -# Extra directories used by stack for building-# extra-include-dirs: [/path/to/dir]-# extra-lib-dirs: [/path/to/dir]-# -# Allow a newer minor version of GHC than the snapshot specifies-# compiler-check: newer-minor+ - '.'+extra-deps:+ - buffer-builder-0.2.4.3
superbuffer.cabal view
@@ -3,7 +3,7 @@ -- see: https://github.com/sol/hpack name: superbuffer-version: 0.1.0.0+version: 0.2.0.0 synopsis: Efficiently build a bytestring from smaller chunks description: Efficiently (both fast and memory efficient) build a bytestring from smaller chunks category: Web@@ -28,7 +28,7 @@ c-sources: cbits/superbuffer.c build-depends:- base >= 4.7 && < 5+ base >= 4.8 && < 5 , bytestring < 0.11 exposed-modules: Data.ByteString.SuperBuffer@@ -44,10 +44,11 @@ ghc-options: -Wall -funfolding-use-threshold=16 -O2 -optc-Ofast cpp-options: -DTest build-depends:- base >= 4.7 && < 5+ base >= 4.8 && < 5 , bytestring < 0.11 , HTF < 0.14 , QuickCheck < 2.10+ , async , superbuffer default-language: Haskell2010 @@ -58,8 +59,10 @@ bench ghc-options: -Wall -funfolding-use-threshold=16 -O2 -optc-Ofast build-depends:- base >= 4.7 && < 5+ base >= 4.8 && < 5 , bytestring < 0.11 , criterion < 1.2 , superbuffer+ , buffer-builder+ , async default-language: Haskell2010
test/Test.hs view
@@ -2,6 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} module Main where +import Control.Concurrent.Async import Control.Monad import Data.ByteString.SuperBuffer import Data.Int@@ -28,6 +29,30 @@ appendBuffer buf " Welcome" appendBuffer buf " to" appendBuffer buf " S U P E R B U F F E R"++test_nullContained :: IO ()+test_nullContained =+ do bs <- fillBuf+ assertEqual bs expected+ where+ expected =+ "hello\0world"+ fillBuf =+ withBuffer 8 $ \buf ->+ do appendBuffer buf "hello"+ appendBuffer buf "\0world"++test_threaded :: IO ()+test_threaded =+ do bs <- fillBuf+ assertEqual bs expected+ where+ expected =+ "hello world! Welcome to S U P E R B U F F E R"+ fillBuf =+ withBuffer 8 $ \buf ->+ forConcurrently_ ["hello", " world", "!", " Welcome", " to", " S U P E R B U F F E R"] $+ appendBufferT buf newtype BufferChunks = BufferChunks { unBufferChunks :: (Int64, [BS.ByteString]) }