Stress-testing remote model querying APIs for relational and graph-based stores

Antonio García-Domínguez*, Konstantinos Barmpis, Dimitrios S. Kolovos, Ran Wei, Richard F. Paige

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Recent research in scalable model-driven engineering now allows very large models to be stored and queried. Due to their size, rather than transferring such models over the network in their entirety, it is typically more efficient to access them remotely using networked services (e.g. model repositories, model indexes). Little attention has been paid so far to the nature of these services, and whether they remain responsive with an increasing number of concurrent clients. This paper extends a previous empirical study on the impact of certain key decisions on the scalability of concurrent model queries on two domains, using an Eclipse Connected Data Objects model repository, four configurations of the Hawk model index and a Neo4j-based configuration of the NeoEMF model store. The study evaluates the impact of the network protocol, the API design, the caching layer, the query language and the type of database and analyses the reasons for their varying levels of performance. The design of the API was shown to make a bigger difference compared to the network protocol (HTTP/TCP) used. Where available, the query-specific indexed and derived attributes in Hawk outperformed the comprehensive generic caching in CDO. Finally, the results illustrate the still ongoing evolution of graph databases: two tools using different versions of the same backend had very different performance, with one slower than CDO and the other faster than it.
Original languageEnglish
Pages (from-to)1047–1075
Number of pages29
JournalSoftware and Systems Modeling
Volume18
Issue number2
Early online date30 Jun 2017
DOIs
Publication statusPublished - 4 Apr 2019

Fingerprint

Application programming interfaces (API)
Testing
Graph in graph theory
Network Protocols
Caching
Model
Repository
Concurrent
Query
Configuration
Network protocols
Object Model
HTTP
Query Language
Query languages
Data Model
Empirical Study
Scalability
Attribute
Engineering

Bibliographical note

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Funding: EPSRC (EP/F001096/1); and EU FP7 (611125).

Keywords

  • API design
  • Collaborative modelling
  • Model persistence
  • NoSQL storage
  • Relational databases
  • Remote model querying
  • Stress testing

Cite this

García-Domínguez, A., Barmpis, K., Kolovos, D. S., Wei, R., & Paige, R. F. (2019). Stress-testing remote model querying APIs for relational and graph-based stores. Software and Systems Modeling, 18(2), 1047–1075. https://doi.org/10.1007/s10270-017-0606-9
García-Domínguez, Antonio ; Barmpis, Konstantinos ; Kolovos, Dimitrios S. ; Wei, Ran ; Paige, Richard F. / Stress-testing remote model querying APIs for relational and graph-based stores. In: Software and Systems Modeling. 2019 ; Vol. 18, No. 2. pp. 1047–1075.
@article{b1c5f33e712f4f25867d61ba0bc44eca,
title = "Stress-testing remote model querying APIs for relational and graph-based stores",
abstract = "Recent research in scalable model-driven engineering now allows very large models to be stored and queried. Due to their size, rather than transferring such models over the network in their entirety, it is typically more efficient to access them remotely using networked services (e.g. model repositories, model indexes). Little attention has been paid so far to the nature of these services, and whether they remain responsive with an increasing number of concurrent clients. This paper extends a previous empirical study on the impact of certain key decisions on the scalability of concurrent model queries on two domains, using an Eclipse Connected Data Objects model repository, four configurations of the Hawk model index and a Neo4j-based configuration of the NeoEMF model store. The study evaluates the impact of the network protocol, the API design, the caching layer, the query language and the type of database and analyses the reasons for their varying levels of performance. The design of the API was shown to make a bigger difference compared to the network protocol (HTTP/TCP) used. Where available, the query-specific indexed and derived attributes in Hawk outperformed the comprehensive generic caching in CDO. Finally, the results illustrate the still ongoing evolution of graph databases: two tools using different versions of the same backend had very different performance, with one slower than CDO and the other faster than it.",
keywords = "API design, Collaborative modelling, Model persistence, NoSQL storage, Relational databases, Remote model querying, Stress testing",
author = "Antonio Garc{\'i}a-Dom{\'i}nguez and Konstantinos Barmpis and Kolovos, {Dimitrios S.} and Ran Wei and Paige, {Richard F.}",
note = "This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Funding: EPSRC (EP/F001096/1); and EU FP7 (611125).",
year = "2019",
month = "4",
day = "4",
doi = "10.1007/s10270-017-0606-9",
language = "English",
volume = "18",
pages = "1047–1075",
number = "2",

}

García-Domínguez, A, Barmpis, K, Kolovos, DS, Wei, R & Paige, RF 2019, 'Stress-testing remote model querying APIs for relational and graph-based stores', Software and Systems Modeling, vol. 18, no. 2, pp. 1047–1075. https://doi.org/10.1007/s10270-017-0606-9

Stress-testing remote model querying APIs for relational and graph-based stores. / García-Domínguez, Antonio; Barmpis, Konstantinos; Kolovos, Dimitrios S.; Wei, Ran; Paige, Richard F.

In: Software and Systems Modeling, Vol. 18, No. 2, 04.04.2019, p. 1047–1075.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Stress-testing remote model querying APIs for relational and graph-based stores

AU - García-Domínguez, Antonio

AU - Barmpis, Konstantinos

AU - Kolovos, Dimitrios S.

AU - Wei, Ran

AU - Paige, Richard F.

N1 - This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Funding: EPSRC (EP/F001096/1); and EU FP7 (611125).

PY - 2019/4/4

Y1 - 2019/4/4

N2 - Recent research in scalable model-driven engineering now allows very large models to be stored and queried. Due to their size, rather than transferring such models over the network in their entirety, it is typically more efficient to access them remotely using networked services (e.g. model repositories, model indexes). Little attention has been paid so far to the nature of these services, and whether they remain responsive with an increasing number of concurrent clients. This paper extends a previous empirical study on the impact of certain key decisions on the scalability of concurrent model queries on two domains, using an Eclipse Connected Data Objects model repository, four configurations of the Hawk model index and a Neo4j-based configuration of the NeoEMF model store. The study evaluates the impact of the network protocol, the API design, the caching layer, the query language and the type of database and analyses the reasons for their varying levels of performance. The design of the API was shown to make a bigger difference compared to the network protocol (HTTP/TCP) used. Where available, the query-specific indexed and derived attributes in Hawk outperformed the comprehensive generic caching in CDO. Finally, the results illustrate the still ongoing evolution of graph databases: two tools using different versions of the same backend had very different performance, with one slower than CDO and the other faster than it.

AB - Recent research in scalable model-driven engineering now allows very large models to be stored and queried. Due to their size, rather than transferring such models over the network in their entirety, it is typically more efficient to access them remotely using networked services (e.g. model repositories, model indexes). Little attention has been paid so far to the nature of these services, and whether they remain responsive with an increasing number of concurrent clients. This paper extends a previous empirical study on the impact of certain key decisions on the scalability of concurrent model queries on two domains, using an Eclipse Connected Data Objects model repository, four configurations of the Hawk model index and a Neo4j-based configuration of the NeoEMF model store. The study evaluates the impact of the network protocol, the API design, the caching layer, the query language and the type of database and analyses the reasons for their varying levels of performance. The design of the API was shown to make a bigger difference compared to the network protocol (HTTP/TCP) used. Where available, the query-specific indexed and derived attributes in Hawk outperformed the comprehensive generic caching in CDO. Finally, the results illustrate the still ongoing evolution of graph databases: two tools using different versions of the same backend had very different performance, with one slower than CDO and the other faster than it.

KW - API design

KW - Collaborative modelling

KW - Model persistence

KW - NoSQL storage

KW - Relational databases

KW - Remote model querying

KW - Stress testing

UR - https://link.springer.com/article/10.1007%2Fs10270-017-0606-9

UR - http://doi.org/10.17036/44783FFA-DA36-424D-9B78-5C3BBAE4AAA1

UR - http://www.scopus.com/inward/record.url?scp=85021753284&partnerID=8YFLogxK

U2 - 10.1007/s10270-017-0606-9

DO - 10.1007/s10270-017-0606-9

M3 - Article

VL - 18

SP - 1047

EP - 1075

IS - 2

ER -

García-Domínguez A, Barmpis K, Kolovos DS, Wei R, Paige RF. Stress-testing remote model querying APIs for relational and graph-based stores. Software and Systems Modeling. 2019 Apr 4;18(2):1047–1075. https://doi.org/10.1007/s10270-017-0606-9