rest,curl,nutch , Nutch 2.3 REST curl syntax

Nutch 2.3 REST curl syntax


Tag: rest,curl,nutch

I'm trying to use curl to test out the Nutch 2.X REST API. I'm able to start the nutchserver and inject URLS, but I'm having trouble getting the generate step to work.

Here's what I've done:

curl -i -X POST -H "Content-Type:application/json" http://localhost:8081/job/create -d '{"crawlId":"crawl-01","type":"INJECT","confId":"default","args":{"seedDir":"/Users/username/myNutchFolder/apache-nutch-2.3/runtime/local/urls/"}}'

which when I look at jobs, shows that it finished and injected the appropriate number of urls.

Then I try to generate using

curl -i -X POST -H "Content-Type:application/json" http://localhost:8081/job/create -d '{"crawlId":"crawl-01","type":"GENERATE","confId":"default","args":{}}'

which fails, and has the following job status:

    "args": {},
    "confId": "default",
    "crawlId": "crawl-01",
    "id": "crawl-01-default-GENERATE-94689123",
    "msg": "ERROR: java.lang.RuntimeException: job failed: name=[crawl-01]generate: null, jobid=job_local473690964_0003",
    "result": null,
    "state": "FAILED",
    "type": "GENERATE"

I can't seem to find any documentation beyond the official API page: so I was hoping someone here might know how to use the REST API to crawl (inject, generate, fetch, parse, updatedb) Any help in understanding even why my generate job failed would be greatly appreciated.


From the user mailing list, I learned the args to use for generate are:







Using .update with nested Serializer to post Image

I have an ImageField. When I update it with the .update command, it does not properly save. It validates, returns a successful save, and says it is good. However, the image is never saved (I don't see it in my /media like I do my other pictures), and when it...

In simple RESTful design, does PATCH imply mapping to CRUD's (ORM's) “update” and PUT to “destroy”+“create” (to replace a resource)?

I'm trying to create a simple REST API and map it to CRUD. I have an ORM (DataMapper) which has methods like create, update and destroy. If I get it right, given a resource {a:'foo',b:'bar',c:'baz'}, performing a PUT {b:'qux'} is supposed to replace the resource and result in the same...

How to use curl return value in php script

I have two servers: A where WP is installed on one, and B where it's not. Application run on B is trying to use WP credentials for login. I have a login form on server B: <h1>Login</h1> <div> <form class="forma" id="form" action="login.php" method="POST"> <div class="form-group"> <label>Username</label> <input class="form-control" type="text" name="username"...

Slow CURL CentOS7 with “same” link

I just installed CentOS7 (3.10.0-229.4.2.el7.x86_64) with nginx (1.8.0). Here my hosts file: [[email protected]_main1 ~]# cat /etc/hosts localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 And problem: [[email protected]_main1 ~]# time curl {"OK"} real 0m0.162s user 0m0.003s sys 0m0.003s [[email protected]_main1 ~]# time curl...

Consuming and exposing webservices in one project (.NET)

What is best practice concerning consuming and exposing webservices in one project? (.net) I need to create a rest webservice to expose data. The rest webservice would need to consume this data from another (SOAP) webservice from a third party. (The data needs to be merged with data present in...

Ruby on Rails - Help Adding Badges to Application

I'm creating a rails application that is a backend for a mobile application. The backend is implemented with a RESTful web API. Currently I am trying to add gamification to the platform through the use of badges that can be earned by the user. Right now the badges are tied...

Trying to write a unit test for file upload to a django Restless API

I'm writing a fairly small lightweight REST api so I chose restless as the quickest/easiest support for that. I didn't seem to need all the complexity and support of the django-REST module. My service will only received and send json but users need to upload files to one single endpoint....

REST Jersey server JAX-RS 500 Internal Server Error

I'm calling this method and getting a 500 back from it. In the debugger I'm able to step though it all the way to the return statement at the end. No problem, r is populated as expected after is called, the status says 200 OK. But that's not what...

Stuck with nested serializer using Django Rest Framework and default user

The models and serializers are described in the pastebin: In my database I have a user which also has a member profile and a set of credentials attached to it. Now... when I run this as is and try to pull a user using the AuthUserModelSerializer I get the...

REST API with token based authentication

I want to develop a web site with AngularJS. On the backend side I will use Codeigniter REST framework. I have some security issues and I don't want to start developing without fixing them on my mind. I don't want to use something like api key because it will be...

incessantly getting null values for reduce function

My data is as follows: { "_id": "33d4d945613344f13a3ee929337b1ca8", "_rev": "1-427c691a5c5f504c6b1d885b6b9ff4bc", "release": { "genres": { "genre": "Electronic" }, "identifiers": { "identifier": [ { "description": "Text", "value": "5 021603 054028", "type": "Barcode" }, { "description": "String", "value": 5021603054028, "type": "Barcode" }, { "value": "MAYKING WAP54CD", "type": "Matrix / Runout" } ] },...

RESTful routing best practice when referencing current_user from route?

I have typical RESTful routes for a user: /user/:id /user/:id/edit /user/:id/newsfeed However the /user/:id/edit route can only be accessed when the id equals the current_user's id. As I only want the current_user to have access to edit its profile. I don't want other users able to edit profiles that don't...

Default/Constant values for POST/PUT arguments with Retrofit

Using the Retrofit REST Client library from Square, is there anyway of providing default/constant values for POST/PUT fields in a call. I know about including constant query parameters by simply including them in the path, but this work for Body parameters. I have an API that looks similar to: POST...

Can't save json data to variable (or cache) with angularjs $http.get

I have weird angularjs problem. I'm trying to fetch data from Rest Webservice. It works fine, but I can't save json data to object. My code looks like: services.service('customerService', [ '$http', '$cacheFactory', function($http, $cacheFactory) { var cache = $cacheFactory('dataCache'); var result = cache.get('user'); this.getById = function(id){ $http.get(urlList.getCustomer + id).success(function(data, status,...

Understanding curl request

I got following line along with the response. How to force that connection is closed after the response? Connection #0 to host localhost left intact curl command: curl -v --user uname:password -H "Accept: application/xml" http://localhost:8090/services/VariableService/variableService/...

How do I silence the HEAD of a curl request while using the silent flag?

When I run the curl command and direct the data to a file, I get back the content of the site as expected. $ curl "" > file.txt $ head file.txt Top of site ... However, this command shows a progress bar, which I do not want: % Total %...

Do we HAVE to generate and use client libraries to use Google App Engine's Endpoints?

I am currently developing an Swift iOS app with GAE Endpoints for the RESTful API. It seems like all the tutorials and documents make you generate and use client libraries if you need to use the API on the client side. I was wondering if it's possible for me to...

XML Post from form using curl PHP

What is the best way to post XML from a form using Curl. I have a HTML Form and i post the data to a new php page and all the fields are collected. How do i collect these fields in XML Format. I can process it from a xml...

Remove resource wrapper from CakePHP REST API JSON

My question is similar to this one. I understand the answer given there. The OP of that question doesn't seem to have my issue. I am using CakePHP 2.2.3. I am fetching a resource like this: http://cakephpsite/lead_posts.json and it returns results like this: [ { "LeadPost": { "id": "1", "fieldA":...

How to manipulate local files with webdav

Hi so I just found out that webdav protocol allows for manipulations of local files through a browser. I have it already set up in the back end. What I would like to know is how to make it work on front end. I am using javascript with jQuery. For...

Sencha/Extjs rest call with all parameters

I'm using ExtJs 5.1.1 and I've written a simple view with a grid, and selecting one row the corresponding model property are editable in some text fields. When editing is completed the button 'save' call method, which use the rest proxy configured to write the changes on the server....


I would like to use CURL in order to APPEND emails at the given date. $ curl -kv -u [email protected]:user 'imap://IP:143' -X 'APPEND INBOX (Mon, 7 Feb > 1994 21:52:25 +0000) {310}' ' Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) From: Fred Foobar <[email protected]> Subject: afternoon meeting To: [email protected]

How to specify supported http operation for a resource in json-ld?

I'm new to JSON-LD and I was wondering if there is any way of specifying supported operation of a resource in JSON-LD without using Hydra's supportedOperation or supportedProperty. Is there any way to specify the context something like : { "@context" : { "@vocab" : "", "data" : "object", "id"...

Unable to upload file to Sharepoint @ Office 365 via REST

I'm having trouble creating/uploading files via Microsoft's REST API (or at least that's what they call it) for Sharepoint running on Office 365. It looks like I'm able to authenticate all right, but I'm getting 403 Forbidden when I try to create a file. The same user can upload a...

json response handling issue

Hi guys I stuck with retrieving json response below is the json output .I novice in this your help would be highly appreciated. { "productHeader" : { "totalHits" : 684 }, "products" : [ { "name" : "Victoria Hotels", "productImage" : { "url" : "" }, "language" : "en", "description"...

Laravel: Retrieve polymorphic attributes efficiently

My actual Question is: How to return attributes to the client which I actually need, throw the rest away and be as efficient as possible? So, I have a construction where I basically have media objects, which can be of different types, like video, picture, link and so on. This...

What's the best way to map objects into ember model from REST Web API?

The topic of this post is: my solution is too slow for a large query return. I have a Web Api serving REST results like below from a call to localhost:9090/api/invetories?id=1: [ { "inventory_id": "1", "film_id": "1", "store_id": "1", "last_update": "2/15/2006 5:09:17 AM" }, { "inventory_id": "2", "film_id": "1", "store_id":...

How to execute POST using CURL

How to execute POST using CURL, i have this basic .asp that receives the data via POST and show the result in .JSON format. <%@LANGUAGE="VBSCRIPT" CODEPAGE="65001"%> <% Response.ContentType = "application/json" Response.Write("{ ""responseCode"": " + Request("responseCode") + ", ""publication_id"": " + Request("publication_id") + ", ""version"": " + Request("version") + "}") %>...

Server-Sent Events Polling causing long delays

I have a connector that will call a RESP API using cURL and PHP. I need to call one method every second to check for new messages and then process them. I used the following 2 approaches to handle the messages AJAX Polling using SetInterval(): call the php script once...

AngularJS $resource Custom Action for Requesting a Password Reset

I'm just starting to use ngResource in a project to consume my RESTful endpoints. Is this how you would implement a user password reset using $resource? Looks weird passing the email address as a URL parameter. .factory('User', ['$resource', function ($resource) { var paramDefaults = {id: '@id'} var actions = {...

Link to another resource in a REST API: by its ID, or by its URL?

I am creating some APIs using apiary, so the language used is JSON. Let's assume I need to represent this resource: { "id" : 9, "name" : "test", "customer_id" : 12, "user_id" : 1, "store_id" : 3, "notes" : "Lorem ipsum example long text" } Is it correct to refer...

python requests with redirection

Trying to authenticate on site, noticed that there were a redirect to Found that there were 302 POST with plain credentials in data form. Copying headers from Chrome can reproduce that in cURL, but still can't reach in requests module. Warning: page is full of russian letters, registration...

REST api : correctly ask for an action

I'm currently working on a REST api. I've read a few times how to handle endpoints the right way, using the protocol (post, put, ...) to define which action should be made. Let's say I have a list of quotes. I have : a GET endpoint /quotes that let me...

Spring Data Rest executes query but returns 500 internal Server Error

I am using spring boot and spring data rest and I am facing a 500 internal server error, but no messages are displayed in console. I have the following: public interface ProdutoVendaRepository extends PagingAndSortingRepository<ProdutoVenda, Integer> { @Query("SELECT new, SUM(i.valorMultiplicado)) FROM ProdutoVenda i JOIN i.produto o JOIN o.familia b...

Springboot REST application should accept and produce both XML and JSON

I am working on Springboot REST API. My application should consume and produce both XML and JSON. I came across the Jackson json Xml dependency. <groupId>com.fasterxml.jackson.dataformat</groupId> <artifactId>jackson-dataformat-xml</artifactId> <version>2.5.4</version> </dependency> I added this in my pom.xml. Now I am able to accept xml input but the values are null when mapped...

Permission denied Setuptools

I'm trying install setuptools in my Mac, but when I run command curl -o - | python show a message telling: Processing setuptools-17.1.1-py3.4.egg Removing /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/setuptools-17.1.1-py3.4.egg Copying setuptools-17.1.1-py3.4.egg to /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages Adding setuptools 17.1.1 to easy-install.pth file error: [Errno 13] Permission denied:...

.NET web service gets null object

I'm trying to save an object using ajax call and .NET REST web service but I get null object on .NET side. Please help me to find out what's the issue. C# Object: [DataContract] public class Customer { [DataMember] public int Id { get; set; } [DataMember] public string Name...

Mailchimp Ecommerce360 Javascript Implementation

Wondering if anyone can provide an example of how to pass a request to the /ecomm/order-add function of the Mailchimp API using javascript. This is critical for making use of Mailchimp's Ecommerce360 tracking. Here is documentation from Mailchimps API:

Office 365 unified api Object reference not set to an instance of an object

I'm trying to use the new office365 unified api to query the users list and user file. I've created the application in azure management portal, and I gave the permission to the new api application (with the directory and files read) I've created both a client and a webapi application,...

Intercepting login calls with Spring-Security-Rest plugin in Grails

I am using the spring security rest plugin for Grails to provide a login mechanism for an AngularJS app. Login works fine, but I can't figure out how to intercept login calls, in order to store additional statistics on (invalid/valid) login attempts. As I am quite new to Spring Security...

What certificates does 'curl' use by default?

What certificates does 'curl' use by default? Example: curl -I -L fails on Ubuntu 15.04 with curl: (60) SSL certificate problem: unable to get local issuer certificate But when I add the root certificate (see and run curl -I -L --cacert downlaodedCert.pem everything is fine. So I...

remote data fetching inside model object in objective c using AFNetworking

In all of my iOS application I use this approach to respect MCV, I want to be sure that my implementation is correct and respects the best practices and the MVC design pattern : Singleton of AFNetworking acting as API for network calls: MyAPI.h : #import "AFHTTPSessionManager.h" #import "AFNetworking.h" @interface...

How can I get json objects without the object number?

I have a simple json object that spits out 4 items that have completely different properties inside each one. I have got the json being displayed with the 4 objects that are called meta.work_content like so: [Object, Object, Object, Object] I can open these in console and see the objects...

How to avoid abusive use of REST endpoint [closed]

how can I avoid abusive use of my REST API? For example, I have a website where certain actions earn a bunch of points which are stored within a user account. So technically, when ever this action is performed, I call my REST endpoint to add the points to the...

CORS, Client vs. Server & Rails API GET Request

I've built a GET Rails API that checks for an access token and that the registered request.env["HTTP_X_REAL_IP"] matches the IP address that is registered within the admin panel of the app. Example request: The API works great when I'm making requests from a server by curling the request or...

Adding authorization to routes

I cannot seem to find a good example for this. I have for example, a TicketController I define a ticket resource in my routes.rb. You only need to be logged in as a customer to GET a ticket, but you must be logged in as an administrator to PUT a...

How to expose existing REST API through Azure Service Bus (or through something else)

I have an existing on-premise REST API from an external vendor. I'd like to expose this API unmodified to the outside world through an Azure website. So I have customers that run this API on-premise and I'm developing a PaaS/SaaS app that should access these on-premise API's. I also have...

@RestController throws HTTP Status 406

I am working on a basic Hello World program using Spring and Restful webservices. But when I try to call my service I am getting below error message: HTTP Status 406 - description - The resource identified by this request is only capable of generating responses with characteristics not acceptable...

How to respond in Middleware Slim PHP Framework

I am creating middleware for auth into REST API. My API is created using Slim PHP Framework ,which in case provide great features to build APIs. One of this feature is Middleware. I need to check credentials in Middleware and respond with an error (HTTP code with JSON descriptions) to...

Unable to select values from the select list

my select list is getting populated via a service call but I cannot select any of the values from the select list. AJS.$("#select2-actor").auiSelect2( { placeholderOption: 'first', formatResult: function(actor) { return '<b>' + actor.text ; }, data: function () { var data = []; AJS.$.ajax({ dataType: 'json', type: 'GET', url: AJS.params.baseURL+"/rest/leangearsrestresource/1.0/message/list/{actor}",...