haskell,web-crawler,monads , How can I get the value of a Monad without System.IO.Unsafe? [duplicate]


How can I get the value of a Monad without System.IO.Unsafe? [duplicate]

Question:

Tag: haskell,web-crawler,monads

This question already has an answer here:

I just started learning Haskell and got my first project working today. Its a small program that uses Network.HTTP.Conduit and Graphics.Rendering.Chart (haskell-chart) to plot the amount of google search results for a specific question with a changing number in it.

My problem is that simple-http from the conduit package returns a monad (I hope I understood the concept of monads right...), but I only want to use the ByteString inside of it, that contains the html-code of the website. So until now i use download = unsafePerformIO $ simpleHttp url to use it later without caring about the monad - I guess that's not the best way to do that.

So: Is there any better solution so that I don't have to carry the monad with me the whole evaluation? Or would it be better to leave it the way the result is returned (with the monad)?

Here's the full program - the mentioned line is in getResultCounter. If things are coded not-so-well and could be done way better, please remark that too:

import System.IO.Unsafe
import Network.HTTP.Conduit (simpleHttp) 
import qualified Data.ByteString.Lazy.Char8 as L
import Graphics.Rendering.Chart.Easy
import Graphics.Rendering.Chart.Backend.Cairo

numchars :: [Char]
numchars = "1234567890"

isNum :: Char -> Bool
isNum = (\x -> x `elem` numchars) 

main = do
    putStrLn "Please input your Search (The first 'X' is going to be replaced): "
    search <- getLine
    putStrLn "X ranges from: "
    from <- getLine
    putStrLn "To: "
    to <- getLine
    putStrLn "In steps of (Only whole numbers are accepted):"
    step <- getLine
    putStrLn "Please have some patience..."
    let range = [read from,(read from + read step)..read to] :: [Int]
    let searches = map (replaceX search) range
    let res = map getResultCounter searches
    plotList search ([(zip range res)] :: [[(Int,Integer)]])
    putStrLn "Done."

-- Creates a plot from the given data
plotList name dat = toFile def (name++".png") $ do
    layout_title .= name
    plot (line "Results" dat)

-- Calls the Google-site and returns the number of results
getResultCounter :: String -> Integer
getResultCounter search = read $ filter isNum $ L.unpack parse :: Integer
    where url = "http://www.google.de/search?q=" ++ search
              download = unsafePerformIO $ simpleHttp url -- Not good 
              parse = takeByteStringUntil "<" 
                      $ dropByteStringUntil "id=\"resultStats\">" download

-- Drops a ByteString until the desired String is found
dropByteStringUntil :: String -> L.ByteString -> L.ByteString
dropByteStringUntil str cont = helper str cont 0
    where helper s bs n | (bs == L.empty) = L.empty
                        | (n >= length s) = bs
                        | ((s !! n) == L.head bs) = helper s (L.tail bs) (n+1)
                        | ((s !! n) /= L.head bs) = helper s (L.tail bs) 0

-- Takes a ByteString until the desired String is found
takeByteStringUntil :: String -> L.ByteString -> L.ByteString
takeByteStringUntil str cont = helper str cont 0
    where helper s bs n | bs == L.empty = bs
                        | n >= length s = L.empty
                        | s !! n == L.head bs = L.head bs `L.cons` 
                                                helper s (L.tail bs) (n + 1)
                        | s !! n /= L.head bs = L.head bs `L.cons` 
                                                helper s (L.tail bs) 0

-- Replaces the first 'X' in a string with the show value of the given value
replaceX :: (Show a) => String -> a -> String
replaceX str x | str == "" = ""
               | head str == 'X' = show x ++ tail str
               | otherwise = head str : replaceX (tail str) x

Answer:

This is a lie:

getResultCounter :: String -> Integer

The type signature above is promising that the resulting integer only depends on the input string, when this is not the case: Google can add/remove results from one call to the other, affecting the output.

Making the type more honest, we get

getResultCounter :: String -> IO Integer

This honestly admits it's going to interact with the external world. The code then is easily adapted to:

getResultCounter search = do
    let url = "http://www.google.de/search?q=" ++ search
    download <- simpleHttp url    -- perform IO here
    let parse = takeByteStringUntil "<" 
                      $ dropByteStringUntil "id=\"resultStats\">" download
    return (read $ filter isNum $ L.unpack parse :: Integer)

Above, I tried to preserve the original structure of the code.

Now, in main we can no longer do

let res = map getResultCounter searches

but we can do

res <- mapM getResultCounter searches

after importing Control.Monad.


Related:


When will travis-ci support ghc 7.10?


haskell,travis-ci
I'd like to use travis-ci on my Haskell projects, but they require the latest version of GHC.

How can I express the type of 'takeWhile for vectors'?


haskell,types,binding,dependent-type
Haskell beginner here. I've defined the following types: data Nat = Z | S Nat data Vector a n where Nil :: Vector a Z (:-) :: a -> Vector a n -> Vector a (S n) infixl 5 :- I'm trying to write the function, takeWhileVector which behaves the...

Setting id and class with the haskell diagrams package


haskell,svg,haskell-diagrams
I am working with the diagrams package for haskell, and I am using the SVG backend. I embed the SVG markup directly into an HTML document, so that the graph as a part of a web page. I have built a pretty cool looking bar graph, and I would like...

Heritrix not finding CSS files in conditional comment blocks


java,web-crawler,heritrix
The Problem/evidence Heritrix is not detecting the presence of files in conditional comments that open & close in one string, such as this: <!--[if (gt IE 8)|!(IE)]><!--> <link rel="stylesheet" href="/css/mod.css" /> <!--<![endif]--> However standard conditional blocks like this work fine: <!--[if lte IE 9]> <script src="/js/ltei9.js"></script> <![endif]--> I've identified the...

Why is lazy evaluation in Haskell “not being lazy”?


haskell,lazy-evaluation
When I tried the following code in cghi: take 1 $ take 1 $ repeat [1..] I was expecting the result of 1 instead of [[1,2,3,4,5,6,7,8,9,10,... printing on my terminal. Why is lazy evaluation not functioning as I'm hoping under such situation?...

How does Frege generalize number literals?


haskell,frege
It appears that Frege can evaluate 1/2 to return the Double value 0.5. The literal 1 is of type Int. It seems to be promoted to Double, which is a type in the Real class and thus knows the / operator. How does this happen? Is it using the Haskell...

Can't find defaultTimeLocale in Data.Time.Format


haskell,cabal
According to the doc on Hackage, Data.Time.Format exposed defaultTimeLocal. However, when I try to use it, it doesn't exits. When I look at the code, it doesn't seems to be exposed either (if I generate the doc from the source, I don't see this defaultTimeLocale). Is it because it needs...

Best practice for handling data types from 3rd party libraries in Haskell?


haskell,types
I'm just getting into my first real Haskell project of size (a web app), and I'm starting to run into issues with types from 3rd party libraries leaking all over my code. Here is a quick example: My Parser module imports Test.Parsec, and the exports a function (parseConfig) that returns...

issues with installing newer cabal version for haskell vim now


ubuntu,haskell,vim,ubuntu-14.04,cabal
I would like to install this vim plugin: https://github.com/begriffs/haskell-vim-now When trying to run the suggested installation script: curl -o - https://raw.githubusercontent.com/begriffs/haskell-vim-now/master/install.sh | bash I get: --- Cabal version 1.18 or later is required. Aborting. I then try to install a newer version of cabal: [email protected]:~/Downloads/cabal-install-1.22.6.0$ ./bootstrap.sh The response I get:...

Thread blocked indefinitely in an MVar operation


haskell,concurrency,network-programming
I have been attempting to debug a problem when using multiple MVars, however to no luck. My code uses two MVars: one to store the servers current state, and another to pass network events to and from the client threads. However after connecting and disconnecting several times, the server stops...

Haskell IO - read from standard input directly to list


haskell
Haskell IO system is super hard to understand for me so i have question : How to read from standard input to list ? I know that there is function getLine :: IO String and interact. But i do not know how to convert the input to list so I...

Implementing map on a tree using fold


scala,haskell
I am trying to implement a map using fold. I could do so in Haskell data Tree a = EmptyTree | Node a (Tree a) (Tree a) deriving (Show) foldTree :: Tree a -> b -> (b -> a -> b -> b) -> b foldTree EmptyTree d _ =...

How to convert a Rational into a “pretty” String?


haskell,formatting,rational
I want to display some Rational values in their decimal expansion. That is, instead of displaying 3 % 4, I would rather display 0.75. I'd like this function to be of type Int -> Rational -> String. The first Int is to specify the maximum number of decimal places, since...

Why is f <$> g <$> x equivalent to (f . g) <$> x although <$> is not right-associative?


haskell,syntax,infix-notation,applicative,infix-operator
Why is f <$> g <$> x equivalent to (f . g) <$> x although <$> is not right-associative? (This kind of equivalence is valid in a popular idiom with plain $, but currently $ is right-associative!) <*> has the same associativity and precedence as <$>, but behaves differently! Example:...

Haskell do clause with multiple monad types


haskell,monads
I'm using a graphic library in Haskell called ThreePennyUI. In this library the main function returns a UI monad object. This causes me much headache as when I attempt to unpack IO values into local variables I receive errors complaining of different monad types. Here's an example of my problem:...

Refactor an IO recursive loop into a monad folding in Haskell


sockets,haskell,network-programming,io-monad
I writing a tcp server, and here's my main loop method: serverLoop :: Socket -> IO () serverLoop sock = do (conn, _) <- accept sock forkIO $ handleConn conn serverLoop sock (Note: handleConn :: Socket -> IO () is a function specific to my program.) I would like to...

Check if element exists in fetched URL [closed]


javascript,jquery,python,web-crawler,window.open
I have a page with, say, 30 URLS, I need to click on each and check if an element exists. Currently, this means: $('area').each(function(){ $(this).attr('target','_blank'); var _href = $(this).attr("href"); var appID = (window.location.href).split('?')[1]; $(this).attr("href", _href + '?' + appID); $(this).trigger('click'); }); Which opens 30 new tabs, and I manually go...

apply a transformation with function inline


haskell
Starting from a simple case of "fold" (I used (+) but can be anything else): Prelude.foldl (+) 0 [10,20,30] is it possible apply an inline transformation similar to (that doesn't work): Prelude.foldl ((+) . (\x -> read x :: Int)) 0 ["10","20","30"] In case not, is there an alternative to...

logical expression evaluator Haskell


haskell
I wrote the following logical expression evaluator. It works for simple 2-member expressions, and it runs but produces a fault for expression containing other expressions as the second/first member. Here's my code. data Expression = Literal Bool | Operation Operator Expression Expression data Operator = AND | OR eval ::...

Get each fibbonacci value in haskell


haskell,fibonacci
I'm learning haskell and I have the following code: fib a b = a : fib b (a + b) findFibSum = sum [x | x <- fib 1 2, mod x 2 == 0 && x < 100] If I run findFibSum nothing happens, it just sits there. Shouldn't...

Idiomatic list construction


list,haskell,functional-programming,idiomatic
I'm very new to Haskell and functional programming in general, so I don't really know how to make this code idiomatic: type Coord = Double data Point = Point Coord Coord Coord deriving Show type Polyline = [Point] -- Add a point to a polyline addPoint :: Polyline -> Point...

Haskell: When declaring a class, how can I use a type variable that is not immediately in the constructors?


haskell
I want to define a function, <-? to check whether an element is in a list/set/map. module Test where import qualified Data.Map as Map import qualified Data.Set as Set class Memberable a where (<-?) :: b -> a -> Bool instance Memberable [x] where (<-?) = elem instance Memberable (Map.Map...

IO Monad Example


haskell
Consider the following IO code: ghci> let x = return 100 :: IO Int ghci> :t do { a <- x; print a; return 500 } do { a <- x; print a; return 500 } :: Num b => IO b My understanding of do notation/bind is that the...

How can I express foldr in terms of foldMap for type-aligned sequences?


haskell,types,monoids,type-variables,foldable
I'm playing around with type-aligned sequences, and in particular I'm messing around with the idea of folding them. A foldable type-aligned sequence looks something like this: class FoldableTA fm where foldMapTA :: Category h => (forall b c . a b c -> h b c) -> fm a b...

Haskell powerset function - How to avoid Couldn't match expected type `IO t0' with actual type `[[Integer]]'


haskell
I have a powerset implementation that I'm trying to run here: http://rextester.com/runcode. I'm still getting this error and can't figure out how to make it right. I'm trying to read as much as possible about IO in haskell but it is super hard for me. import Control.Monad (filterM) powerset =...

Fold over a heterogeneous, compile time, list


haskell,type-level-computation,hlist
I have a list of heterogeneous types (or at least that's what I have in mind): data Nul data Bits b otherBits where BitsLst :: b -> otherBits -> Bits b otherBits NoMoreBits :: Bits b Nul Now, given an input type b, I want to go through all the...

Haskell - generate and use the same random list


haskell,random
In Haskell, I'd like to generate a random list of Ints and use it throughout my program. My current solution causes the array to be created randomly each time I access/use it. How can I overcome this problem?...

Distinguishing between HTML and non-HTML pages in Scrapy


python,html,web-crawler,scrapy,scrapy-spider
I am building a Spider in Scrapy that follows all the links it can find, and sends the url to a pipeline. At the moment, this is my code: from scrapy import Spider from scrapy.http import Request from scrapy.http import TextResponse from scrapy.selector import Selector from scrapyTest.items import TestItem import...

Keep track of loop without a counter


haskell
Say, I got a list which length can be odd or even. Every iteration , I remove two items from the list. If there is one or no item at the end, I end the execution. If I store (length list)/2 every loop, I will get e.g. [5,4,3...] for a...

Replace all [ ] with {} - as short as possible [on hold]


haskell
Given the code below: import Data.List; main = (readLn :: IO [Integer]) >>= print . subsequences It takes a list of integers from standard input (for example [1,2,3]) and outputs something like: [[],[1],[2],[1,2],[3],[1,3],[2,3],[1,2,3]] I want it to be like this: {},{1},{2},{1,2},{3},{1,3},{2,3},{1,2,3}} so my goal is to replace every [ and...

Tokenizer identifier in Haskell


haskell
I'm writing this small program basically to identify each input tokens as operator/parenthesis/int. However, I encountered a problem stating that Not in scope: data constructor `Integer' Here's what I have so far (Data.Char only defines isDigit, nothing else) import Data.Char (isDigit) data Token = TPlus | TTimes | TParenLeft |...

Running executable files using Haskell


shell,haskell,command-line-arguments,executable
Say there's a C++ code that I've compiled into an executable using: g++ test.cpp -o testcpp I can run this using Terminal (I'm using OS X), and provide an input file for processing inside the C++ program, like: ./testcpp < input.txt I was wondering if doing this is possible, from...

T_STRING error in my php code [duplicate]


php,web-crawler
This question already has an answer here: PHP Parse/Syntax Errors; and How to solve them? 10 answers I have this PHP that is supposed to crawl End Clothing website for product IDs When I run it its gives me this error Parse error: syntax error, unexpected 'i' (T_STRING), expecting...

Why doesn't `iterate` from the Prelude tie the knot?


haskell,tying-the-knot
Why isn't iterate defined like iterate :: (a -> a) -> a -> [a] iterate f x = xs where xs = x : map f xs in the Prelude?...

Hook into GHC runtime system


haskell,functional-programming,runtime,transactional-memory
I have been looking at how transactional memory is implemented in Haskell, and I am not sure I understand how the STM operations exposed to the programmer hook into the runtime system functions written in C. In ghc/libraries/base/GHC/Conc/Sync.hs of the git repo, I see the following definitions: -- |A monad...

Stopping condition on a recursive function - Haskell


string,function,haskell,if-statement,recursion
So, I have this function which aims to align the text on the left without cutting words(only white spaces). However my problem is that I cannot find a stopping condition of the function and it goes infinitely. f n "" = "" --weak condition f n s = if n...

How to “wrap” monadic return value


haskell,monads
I have the following code: data APNSIdentifier = NoIdentifier | Identifier Word32 deriving (Show, Eq) newtype APNSItem = Item Put createNotificationIdentifierItem :: APNSIdentifier -> APNSItem <--- here createNotificationIdentifierItem (Identifier identifier) = do putWord8 3 putWord16be 4 putWord32be identifier How can I "wrap" the Put monad with an APNSItem? Do I...

Recursion scheme in Haskell for repeatedly breaking datatypes into “head” and “tail” and yielding a structure of results


haskell,recursion
In Haskell, I recently found the following function useful: listCase :: (a -> [a] -> b) -> [a] -> [b] listCase f [] = [] listCase f (x:xs) = f x xs : listCase f xs I used it to generate sliding windows of size 3 from a list, like...

Combining Event and an attribute in threepenny-gui


haskell,threepenny-gui
I have an Event String which I want to sink into a textarea. This works fine, but now I want to combine the current value of a checkbox selection to this string. First I did this by using checkedChange of the checkbox which works, but has a problem. If the...

Haskell make recipe fails for Paradox theorem prover using GHC


linux,haskell,make,ghc,theorem-proving
I am trying to install the paradox theorem prover sourced from here. When I run the makefile this is the command that runs: ghc -optl -static -lstdc++ -I../instantiate -I../minisat/current-base ../minisat/current-base/Solver.or ../minisat/current-base/Prop.or ../instantiate/MiniSatWrapper.or ../instantiate/MiniSatInstantiateClause.or -fglasgow-exts -O2 -static -threaded -main-is Paradox.Main.main --make Paradox.Main -o paradox And it results in several errors like...

Scrapy not entering parse method


python,selenium,web-scraping,web-crawler,scrapy
I don't understand why this code is not entering the parse method. It is pretty similar to the basic spider examples from the doc: http://doc.scrapy.org/en/latest/topics/spiders.html And I'm pretty sure this worked earlier in the day... Not sure if I modified something or not.. from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.common.by import...

Mapping with IO actions in Haskell


list,haskell,io
I have a function that takes two filenames, and reads the contents of those two files into Strings, and then returns if they match or not. Here's the function: f :: String -> String -> IO Bool f fileName1 fileName2 = do str1 <- readFile fileName1 str2 <- readFile fileName2...

Apache Nutch REST api


api,rest,web-crawler,nutch
I'm trying to launch a crawl via the rest api. A crawl starts with injecting urls. Using a chrome developer tool "Advanced Rest Client" I'm trying to build this POST payload up but the response I get is a 400 Bad Request. POST - http://localhost:8081/job/create Payload { "crawl-id":"crawl-01", "type":"INJECT", "config-id":"default",...

Decremented value called in the recursion in Haskell


string,function,haskell,recursion,parameters
So, I have this function that aligns the text to the left for a given column width. I have already fixed some of its problems, but I am unable to make it not use the decremented parameter once it is called, but to return to the starting value of n....

Python 3.3 TypeError: can't use a string pattern on a bytes-like object in re.findall()


python-3.x,web-crawler
I am trying to learn how to automatically fetch urls from a page. In the following code I am trying to get the title of the webpage: import urllib.request import re url = "http://www.google.com" regex = '<title>(,+?)</title>' pattern = re.compile(regex) with urllib.request.urlopen(url) as response: html = response.read() title = re.findall(pattern,...

Normal probability density function - GSL equivalent in Haskell


haskell,statistics,gsl
I am trying to find the best way to draw from a normal distribution. I want to be able to use the normal probability density function (and its cumulative) in Haskell. To put it simply, I want to use the functions on this page without using the GSL binding... I...

What is haskellng? What is the difference between 'haskellPackages' and 'haskellngPackages'?


haskell,cabal,cabal-install,nix,haskell-ng
I have been reading this StackOverflow post in which we are advised to use the haskellng package set. I have also read this but I did not understand what haskellng is. I have read this too, but I still don't know what haskellng is. Could someone please explain what haskellng...

First three items of a list in Haskell


haskell
I am very new to Haskell, and struggling a bit with a function here. The premise is simple enough: Run through a list, and combine each 3 items next to each other with another function and return a list with the results. The problem is to do it in a...

How do I avoid writing this type of Haskell boilerplate code


haskell,boilerplate
I run into this situation often enough for it to be annoying. Let's say I have a sum type which can hold an instance of x or a bunch of other things unrelated to x - data Foo x = X x | Y Int | Z String | ...(other...

Haskell return lazy string from file IO


haskell,file-io,lazy-evaluation
Here I'm back again with a (for me) really strange behaviour of my newest masterpiece... This code should read a file, but it doesn't: readCsvContents :: String -> IO ( String ) readCsvContents fileName = do withFile fileName ReadMode (\handle -> do contents <- hGetContents handle return contents ) main...