rest,curl,nutch , Nutch 2.3 REST curl syntax


Nutch 2.3 REST curl syntax

Question:

Tag: rest,curl,nutch

I'm trying to use curl to test out the Nutch 2.X REST API. I'm able to start the nutchserver and inject URLS, but I'm having trouble getting the generate step to work.

Here's what I've done:

curl -i -X POST -H "Content-Type:application/json" http://localhost:8081/job/create -d '{"crawlId":"crawl-01","type":"INJECT","confId":"default","args":{"seedDir":"/Users/username/myNutchFolder/apache-nutch-2.3/runtime/local/urls/"}}'

which when I look at jobs, shows that it finished and injected the appropriate number of urls.

Then I try to generate using

curl -i -X POST -H "Content-Type:application/json" http://localhost:8081/job/create -d '{"crawlId":"crawl-01","type":"GENERATE","confId":"default","args":{}}'

which fails, and has the following job status:

{
    "args": {},
    "confId": "default",
    "crawlId": "crawl-01",
    "id": "crawl-01-default-GENERATE-94689123",
    "msg": "ERROR: java.lang.RuntimeException: job failed: name=[crawl-01]generate: null, jobid=job_local473690964_0003",
    "result": null,
    "state": "FAILED",
    "type": "GENERATE"
},

I can't seem to find any documentation beyond the official API page: https://wiki.apache.org/nutch/NutchRESTAPI#Create_job so I was hoping someone here might know how to use the REST API to crawl (inject, generate, fetch, parse, updatedb) Any help in understanding even why my generate job failed would be greatly appreciated.


Answer:

From the user mailing list, I learned the args to use for generate are:

"normalize":boolean

"filter":boolean

"crawlId":String

"curTime":long

"batch":String


Related:


incessantly getting null values for reduce function


curl,couchdb
My data is as follows: { "_id": "33d4d945613344f13a3ee929337b1ca8", "_rev": "1-427c691a5c5f504c6b1d885b6b9ff4bc", "release": { "genres": { "genre": "Electronic" }, "identifiers": { "identifier": [ { "description": "Text", "value": "5 021603 054028", "type": "Barcode" }, { "description": "String", "value": 5021603054028, "type": "Barcode" }, { "value": "MAYKING WAP54CD", "type": "Matrix / Runout" } ] },...

Intercepting login calls with Spring-Security-Rest plugin in Grails


rest,grails,spring-security
I am using the spring security rest plugin for Grails to provide a login mechanism for an AngularJS app. Login works fine, but I can't figure out how to intercept login calls, in order to store additional statistics on (invalid/valid) login attempts. As I am quite new to Spring Security...

REST api : correctly ask for an action


api,rest,endpoint
I'm currently working on a REST api. I've read a few times how to handle endpoints the right way, using the protocol (post, put, ...) to define which action should be made. Let's say I have a list of quotes. I have : a GET endpoint /quotes that let me...

REST API with token based authentication


angularjs,codeigniter,api,rest,token
I want to develop a web site with AngularJS. On the backend side I will use Codeigniter REST framework. I have some security issues and I don't want to start developing without fixing them on my mind. I don't want to use something like api key because it will be...

How to specify supported http operation for a resource in json-ld?


rest,http-method,json-ld
I'm new to JSON-LD and I was wondering if there is any way of specifying supported operation of a resource in JSON-LD without using Hydra's supportedOperation or supportedProperty. Is there any way to specify the context something like : { "@context" : { "@vocab" : "http://www.schema.org/", "data" : "object", "id"...

Trying to write a unit test for file upload to a django Restless API


python,django,rest,file-upload,request
I'm writing a fairly small lightweight REST api so I chose restless as the quickest/easiest support for that. I didn't seem to need all the complexity and support of the django-REST module. My service will only received and send json but users need to upload files to one single endpoint....

Laravel: Retrieve polymorphic attributes efficiently


rest,laravel,polymorphism,eloquent
My actual Question is: How to return attributes to the client which I actually need, throw the rest away and be as efficient as possible? So, I have a construction where I basically have media objects, which can be of different types, like video, picture, link and so on. This...

Using .update with nested Serializer to post Image


django,rest,django-models,django-rest-framework,imagefield
I have an ImageField. When I update it with the .update command, it does not properly save. It validates, returns a successful save, and says it is good. However, the image is never saved (I don't see it in my /media like I do my other pictures), and when it...

Slow CURL CentOS7 with “same” link


curl,dns,centos,hosts
I just installed CentOS7 (3.10.0-229.4.2.el7.x86_64) with nginx (1.8.0). Here my hosts file: [[email protected]_main1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 arm.site.com kpp.site.com w.site.com server.site.com And problem: [[email protected]_main1 ~]# time curl http://arm.site.com/test/fad/site/site?siteId=152 {"OK"} real 0m0.162s user 0m0.003s sys 0m0.003s [[email protected]_main1 ~]# time curl...

Sencha/Extjs rest call with all parameters


json,rest,extjs,sencha-touch
I'm using ExtJs 5.1.1 and I've written a simple view with a grid, and selecting one row the corresponding model property are editable in some text fields. When editing is completed the button 'save' call Model.save() method, which use the rest proxy configured to write the changes on the server....

REST Jersey server JAX-RS 500 Internal Server Error


java,rest,jersey,jax-rs
I'm calling this method and getting a 500 back from it. In the debugger I'm able to step though it all the way to the return statement at the end. No problem, r is populated as expected after Response.build() is called, the status says 200 OK. But that's not what...

Default/Constant values for POST/PUT arguments with Retrofit


java,rest,retrofit
Using the Retrofit REST Client library from Square, is there anyway of providing default/constant values for POST/PUT fields in a call. I know about including constant query parameters by simply including them in the path, but this work for Body parameters. I have an API that looks similar to: POST...

How to manipulate local files with webdav


javascript,jquery,rest,file-upload,webdav
Hi so I just found out that webdav protocol allows for manipulations of local files through a browser. I have it already set up in the back end. What I would like to know is how to make it work on front end. I am using javascript with jQuery. For...

CORS, Client vs. Server & Rails API GET Request


ruby-on-rails,ruby,api,curl,client
I've built a GET Rails API that checks for an access token and that the registered request.env["HTTP_X_REAL_IP"] matches the IP address that is registered within the admin panel of the app. Example request: https://staging.mysite.com/api/v1/products?access_token=7b9f3cddd3914a6f45fa692997fe6dc9 The API works great when I'm making requests from a server by curling the request or...

Link to another resource in a REST API: by its ID, or by its URL?


json,api,rest,api-design,hateoas
I am creating some APIs using apiary, so the language used is JSON. Let's assume I need to represent this resource: { "id" : 9, "name" : "test", "customer_id" : 12, "user_id" : 1, "store_id" : 3, "notes" : "Lorem ipsum example long text" } Is it correct to refer...

Do we HAVE to generate and use client libraries to use Google App Engine's Endpoints?


ios,swift,rest,google-app-engine,google-cloud-endpoints
I am currently developing an Swift iOS app with GAE Endpoints for the RESTful API. It seems like all the tutorials and documents make you generate and use client libraries if you need to use the API on the client side. I was wondering if it's possible for me to...

json response handling issue


php,json,curl
Hi guys I stuck with retrieving json response below is the json output .I novice in this your help would be highly appreciated. { "productHeader" : { "totalHits" : 684 }, "products" : [ { "name" : "Victoria Hotels", "productImage" : { "url" : "http://hotels.com/hotels/9000000/8640000/8633700/8633672/8633672_20_b.jpg" }, "language" : "en", "description"...

Unable to upload file to Sharepoint @ Office 365 via REST


javascript,ajax,rest,sharepoint,office365
I'm having trouble creating/uploading files via Microsoft's REST API (or at least that's what they call it) for Sharepoint running on Office 365. It looks like I'm able to authenticate all right, but I'm getting 403 Forbidden when I try to create a file. The same user can upload a...

Unable to select values from the select list


javascript,jquery,rest
my select list is getting populated via a service call but I cannot select any of the values from the select list. AJS.$("#select2-actor").auiSelect2( { placeholderOption: 'first', formatResult: function(actor) { return '<b>' + actor.text ; }, data: function () { var data = []; AJS.$.ajax({ dataType: 'json', type: 'GET', url: AJS.params.baseURL+"/rest/leangearsrestresource/1.0/message/list/{actor}",...

.NET web service gets null object


c#,.net,ajax,web-services,rest
I'm trying to save an object using ajax call and .NET REST web service but I get null object on .NET side. Please help me to find out what's the issue. C# Object: [DataContract] public class Customer { [DataMember] public int Id { get; set; } [DataMember] public string Name...

How to use curl return value in php script


php,wordpress,curl,login
I have two servers: A where WP is installed on one, and B where it's not. Application run on B is trying to use WP credentials for login. I have a login form on server B: <h1>Login</h1> <div> <form class="forma" id="form" action="login.php" method="POST"> <div class="form-group"> <label>Username</label> <input class="form-control" type="text" name="username"...

Ruby on Rails - Help Adding Badges to Application


ruby-on-rails,ruby,rest,activerecord,one-to-many
I'm creating a rails application that is a backend for a mobile application. The backend is implemented with a RESTful web API. Currently I am trying to add gamification to the platform through the use of badges that can be earned by the user. Right now the badges are tied...

Stuck with nested serializer using Django Rest Framework and default user


django,api,rest,django-rest-framework,serializer
The models and serializers are described in the pastebin: http://pastebin.com/ZxzxWY7V In my database I have a user which also has a member profile and a set of credentials attached to it. Now... when I run this as is and try to pull a user using the AuthUserModelSerializer I get the...

Can't save json data to variable (or cache) with angularjs $http.get


json,angularjs,web-services,rest
I have weird angularjs problem. I'm trying to fetch data from Rest Webservice. It works fine, but I can't save json data to object. My code looks like: services.service('customerService', [ '$http', '$cacheFactory', function($http, $cacheFactory) { var cache = $cacheFactory('dataCache'); var result = cache.get('user'); this.getById = function(id){ $http.get(urlList.getCustomer + id).success(function(data, status,...

python requests with redirection


python,authentication,redirect,curl,python-requests
Trying to authenticate on http://72.ru site, noticed that there were a redirect to https://loginka.ru/auth/. Found that there were 302 POST with plain credentials in data form. Copying headers from Chrome can reproduce that in cURL, but still can't reach in requests module. Warning: page is full of russian letters, registration...

RESTful routing best practice when referencing current_user from route?


ruby-on-rails,rest
I have typical RESTful routes for a user: /user/:id /user/:id/edit /user/:id/newsfeed However the /user/:id/edit route can only be accessed when the id equals the current_user's id. As I only want the current_user to have access to edit its profile. I don't want other users able to edit profiles that don't...

Adding authorization to routes


ruby-on-rails,rest,routes,authorization
I cannot seem to find a good example for this. I have for example, a TicketController I define a ticket resource in my routes.rb. You only need to be logged in as a customer to GET a ticket, but you must be logged in as an administrator to PUT a...

How to expose existing REST API through Azure Service Bus (or through something else)


rest,azure,azureservicebus
I have an existing on-premise REST API from an external vendor. I'd like to expose this API unmodified to the outside world through an Azure website. So I have customers that run this API on-premise and I'm developing a PaaS/SaaS app that should access these on-premise API's. I also have...

What's the best way to map objects into ember model from REST Web API?


json,rest,ember.js,asp.net-web-api,ember-data
The topic of this post is: my solution is too slow for a large query return. I have a Web Api serving REST results like below from a call to localhost:9090/api/invetories?id=1: [ { "inventory_id": "1", "film_id": "1", "store_id": "1", "last_update": "2/15/2006 5:09:17 AM" }, { "inventory_id": "2", "film_id": "1", "store_id":...

Spring Data Rest executes query but returns 500 internal Server Error


java,spring,rest,spring-boot,spring-data-rest
I am using spring boot and spring data rest and I am facing a 500 internal server error, but no messages are displayed in console. I have the following: ProdutoVendaRepository.java public interface ProdutoVendaRepository extends PagingAndSortingRepository<ProdutoVenda, Integer> { @Query("SELECT new br.com.contoso.model.VendaPorFamilia(b.nome, SUM(i.valorMultiplicado)) FROM ProdutoVenda i JOIN i.produto o JOIN o.familia b...

Office 365 unified api Object reference not set to an instance of an object


php,curl,header,office365
I'm trying to use the new office365 unified api to query the users list and user file. I've created the application in azure management portal, and I gave the permission to the new api application (with the directory and files read) I've created both a client and a webapi application,...

AngularJS $resource Custom Action for Requesting a Password Reset


angularjs,rest,ngresource,angularjs-1.3
I'm just starting to use ngResource in a project to consume my RESTful endpoints. Is this how you would implement a user password reset using $resource? Looks weird passing the email address as a URL parameter. .factory('User', ['$resource', function ($resource) { var paramDefaults = {id: '@id'} var actions = {...

How to respond in Middleware Slim PHP Framework


php,rest,authentication,middleware,slim
I am creating middleware for auth into REST API. My API is created using Slim PHP Framework ,which in case provide great features to build APIs. One of this feature is Middleware. I need to check credentials in Middleware and respond with an error (HTTP code with JSON descriptions) to...

@RestController throws HTTP Status 406


java,spring,rest,maven
I am working on a basic Hello World program using Spring and Restful webservices. But when I try to call my service I am getting below error message: HTTP Status 406 - description - The resource identified by this request is only capable of generating responses with characteristics not acceptable...

CURL IMAP APPEND command


curl,imap
I would like to use CURL in order to APPEND emails at the given date. $ curl -kv -u [email protected]:user 'imap://IP:143' -X 'APPEND INBOX (Mon, 7 Feb > 1994 21:52:25 +0000) {310}' ' Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) From: Fred Foobar <[email protected]> Subject: afternoon meeting To: [email protected]

How can I get json objects without the object number?


javascript,jquery,json,rest
I have a simple json object that spits out 4 items that have completely different properties inside each one. I have got the json being displayed with the 4 objects that are called meta.work_content like so: [Object, Object, Object, Object] I can open these in console and see the objects...

Springboot REST application should accept and produce both XML and JSON


java,xml,rest,jackson,spring-boot
I am working on Springboot REST API. My application should consume and produce both XML and JSON. I came across the Jackson json Xml dependency. <groupId>com.fasterxml.jackson.dataformat</groupId> <artifactId>jackson-dataformat-xml</artifactId> <version>2.5.4</version> </dependency> I added this in my pom.xml. Now I am able to accept xml input but the values are null when mapped...

Permission denied Setuptools


python,django,curl,setuptools
I'm trying install setuptools in my Mac, but when I run command curl https://bootstrap.pypa.io/ez_setup.py -o - | python show a message telling: Processing setuptools-17.1.1-py3.4.egg Removing /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/setuptools-17.1.1-py3.4.egg Copying setuptools-17.1.1-py3.4.egg to /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages Adding setuptools 17.1.1 to easy-install.pth file error: [Errno 13] Permission denied:...

How to execute POST using CURL


post,curl,asp-classic
How to execute POST using CURL, i have this basic .asp that receives the data via POST and show the result in .JSON format. <%@LANGUAGE="VBSCRIPT" CODEPAGE="65001"%> <% Response.ContentType = "application/json" Response.Write("{ ""responseCode"": " + Request("responseCode") + ", ""publication_id"": " + Request("publication_id") + ", ""version"": " + Request("version") + "}") %>...

What certificates does 'curl' use by default?


curl,certificate
What certificates does 'curl' use by default? Example: curl -I -L https://cruises.webjet.com.au fails on Ubuntu 15.04 with curl: (60) SSL certificate problem: unable to get local issuer certificate But when I add the root certificate (see https://www.ssllabs.com/ssltest/analyze.html?d=cruises.webjet.com.au&latest) and run curl -I -L --cacert downlaodedCert.pem https://cruises.webjet.com.au everything is fine. So I...

Server-Sent Events Polling causing long delays


javascript,php,jquery,curl,server-sent-events
I have a connector that will call a RESP API using cURL and PHP. I need to call one method every second to check for new messages and then process them. I used the following 2 approaches to handle the messages AJAX Polling using SetInterval(): call the php script once...

How to avoid abusive use of REST endpoint [closed]


java,javascript,rest
how can I avoid abusive use of my REST API? For example, I have a website where certain actions earn a bunch of points which are stored within a user account. So technically, when ever this action is performed, I call my REST endpoint to add the points to the...

In simple RESTful design, does PATCH imply mapping to CRUD's (ORM's) “update” and PUT to “destroy”+“create” (to replace a resource)?


database,rest,http,orm,crud
I'm trying to create a simple REST API and map it to CRUD. I have an ORM (DataMapper) which has methods like create, update and destroy. If I get it right, given a resource {a:'foo',b:'bar',c:'baz'}, performing a PUT {b:'qux'} is supposed to replace the resource and result in the same...

Consuming and exposing webservices in one project (.NET)


.net,web-services,rest,soap
What is best practice concerning consuming and exposing webservices in one project? (.net) I need to create a rest webservice to expose data. The rest webservice would need to consume this data from another (SOAP) webservice from a third party. (The data needs to be merged with data present in...

Remove resource wrapper from CakePHP REST API JSON


rest,cakephp,cakephp-2.2
My question is similar to this one. I understand the answer given there. The OP of that question doesn't seem to have my issue. I am using CakePHP 2.2.3. I am fetching a resource like this: http://cakephpsite/lead_posts.json and it returns results like this: [ { "LeadPost": { "id": "1", "fieldA":...

XML Post from form using curl PHP


php,xml,curl
What is the best way to post XML from a form using Curl. I have a HTML Form and i post the data to a new php page and all the fields are collected. How do i collect these fields in XML Format. I can process it from a xml...

Mailchimp Ecommerce360 Javascript Implementation


javascript,rest,e-commerce,mailchimp
Wondering if anyone can provide an example of how to pass a request to the /ecomm/order-add function of the Mailchimp API using javascript. This is critical for making use of Mailchimp's Ecommerce360 tracking. Here is documentation from Mailchimps API: https://apidocs.mailchimp.com/api/2.0/ecomm/order-add.php...

How do I silence the HEAD of a curl request while using the silent flag?


bash,shell,curl,command-line,pipe
When I run the curl command and direct the data to a file, I get back the content of the site as expected. $ curl "www.site.com" > file.txt $ head file.txt Top of site ... However, this command shows a progress bar, which I do not want: % Total %...

remote data fetching inside model object in objective c using AFNetworking


ios,objective-c,rest,model-view-controller,afnetworking-2
In all of my iOS application I use this approach to respect MCV, I want to be sure that my implementation is correct and respects the best practices and the MVC design pattern : Singleton of AFNetworking acting as API for network calls: MyAPI.h : #import "AFHTTPSessionManager.h" #import "AFNetworking.h" @interface...

Understanding curl request


curl
I got following line along with the response. How to force that connection is closed after the response? Connection #0 to host localhost left intact curl command: curl -v --user uname:password -H "Accept: application/xml" http://localhost:8090/services/VariableService/variableService/...